The above contains links to a lot of arguments, but it does not develop almost any argument fully.
In lieu of responding to them all, I will address one sentence.
Here’s a claim you make about likely future timelines: “GPT-4 + curious (but ultimately reckless) academics → more efficient AI → next generation foundation model AI (which I’ll call NextAI[10] for short)”
There are two links in “ultimately reckless” go to two papers, which must be to support the claim that researchers are reckless, or the claim that we will quickly get more efficient next level AI, presumably.
One paper uses GPT-4 to do neural architecture search over CIFAR-10, or CIFAR-100, or ImageNet16-120. These are all pretty tiny datasets… which is why people do NAS over them, because otherwise it would be insanely expensive. Even doing this kind of thing with a small LM would be really tough. Furthermore, GPT-4 looks like it’s giving you the same approximate OOM improvements that other NAS techniques do, which isn’t that great. Like I could write more here, but this is basically miles away from being used to help with actual LLMs, let alone recursive self improvement—it’s on toy datasets used for NAS because they are small.
The other uses GPT-4 augmented with planners to produce plans. “Plans” means classical planing problems like “Given a set of piles of blocks on a table, a robot is tasked with rearranging them into a specified target configuration while obeying the laws of physics.” This kind of thing—with definite ontology, non-vague world, etc—has been in AI since the 70s and the ability of GPT-4 to work with a classical planner is basically entirely unrelated to… lengthy, contextually vague problems necessary to take over the world. Like, it isn’t just that “Take over the world” is too vague and onto logically unspecified for this kind of treatment, “Unload my car, don’t break anything” is also too vague and unspecified. In any event, I don’t think it’s remotely worrisome, and in any event extremely unclear how it leads to more efficient NextAI.
Anyhow, neither of these papers lead me to think that researchers are reckless. I’m really unsure how they’re suppose to support that claim, or the claim that we’ll get NextAI quickly, or even what NextAI would be.
Anyhow, if the above sentence is representative, the above looks like a gish gallop—responding at length to many of these points would take 50x the space spent here.
So Gish gallop is not the ideal phrasing, although denotatively that is what I think it is.
A more productive phrasing on my part would be, when arguing it is charitable to only put forth the strongest arguments you have, rather than many questionable arguments.
This helps you persuade other people, if you are right, because they won’t see a weaker argument and think all your arguments are that weak.
This helps you be corrected, if you are wrong, because it’s more likely that someone will be able to respond to one or two arguments that you have identified as strong arguments, and show you where you are wrong—no one’s going to do that with 20 weaker arguments, because who has the time?
Put alternately, it also helps with epistemic legibility. It also shows that you aren’t just piling up a lot of soldiers for your side—it shows that you’ve put in the work to weed out the ones which matter, and are not putting weird demands on your reader’s attention by just putting all those which work.
You have a lot of sub-parts in your argument for 1, 2, 3 above. (Like, in the first section there are ~5 points I think are just wrong or misleading, and regardless of whether they are wrong or not are at least highly disputed). It doesn’t help to have succession of such disputed points—regardless of whether your audience is people you agree with or people you don’t!
The way I see the above post (and it’s accompaniment) is knocking down all the soldiers that I’ve encountered talking to lots of people about this over the last few weeks. I would appreciate it if you could stand them back up (because I’m reallytrying to not be so doomy, and not getting any satisfactory rebuttals).
I think you need to zoom out a bit and look at the implications of these papers. The danger isn’t in what people are doing now, it’s in what they might be doing in a few months following on from this work. The NAS paper was a proof of concept. What happens when it’s massively scaled up? What happens when efficiency gains translate into further efficiency gains?
Here’s a chart of one of the benchmarks the GPT-NAS paper tests on. They GPT-NAS paper is like.… not off trend? Not even SOTA? Honestly looking at all these results my tenative guess is that the differences are basically noise for most techniques; the state space is tiny such that I doubt any of these really leverage actual regularities in it.
The above contains links to a lot of arguments, but it does not develop almost any argument fully.
In lieu of responding to them all, I will address one sentence.
Here’s a claim you make about likely future timelines: “GPT-4 + curious (but ultimately reckless) academics → more efficient AI → next generation foundation model AI (which I’ll call NextAI[10] for short)”
There are two links in “ultimately reckless” go to two papers, which must be to support the claim that researchers are reckless, or the claim that we will quickly get more efficient next level AI, presumably.
One paper uses GPT-4 to do neural architecture search over CIFAR-10, or CIFAR-100, or ImageNet16-120. These are all pretty tiny datasets… which is why people do NAS over them, because otherwise it would be insanely expensive. Even doing this kind of thing with a small LM would be really tough. Furthermore, GPT-4 looks like it’s giving you the same approximate OOM improvements that other NAS techniques do, which isn’t that great. Like I could write more here, but this is basically miles away from being used to help with actual LLMs, let alone recursive self improvement—it’s on toy datasets used for NAS because they are small.
The other uses GPT-4 augmented with planners to produce plans. “Plans” means classical planing problems like “Given a set of piles of blocks on a table, a robot is tasked with rearranging them into a specified target configuration while obeying the laws of physics.” This kind of thing—with definite ontology, non-vague world, etc—has been in AI since the 70s and the ability of GPT-4 to work with a classical planner is basically entirely unrelated to… lengthy, contextually vague problems necessary to take over the world. Like, it isn’t just that “Take over the world” is too vague and onto logically unspecified for this kind of treatment, “Unload my car, don’t break anything” is also too vague and unspecified. In any event, I don’t think it’s remotely worrisome, and in any event extremely unclear how it leads to more efficient NextAI.
Anyhow, neither of these papers lead me to think that researchers are reckless. I’m really unsure how they’re suppose to support that claim, or the claim that we’ll get NextAI quickly, or even what NextAI would be.
Anyhow, if the above sentence is representative, the above looks like a gish gallop—responding at length to many of these points would take 50x the space spent here.
It’s really not intended as a gish gallop, sorry if you are seeing it as such. I feel like I’m really only making 3 arguments:
1. AGI is near
2. Alignment isn’t ready (and therefore P(doom|AGI is high)
3. AGI is dangerous
And then drawing the conclusion from all these that we need a global AGI moratorium asap.
So Gish gallop is not the ideal phrasing, although denotatively that is what I think it is.
A more productive phrasing on my part would be, when arguing it is charitable to only put forth the strongest arguments you have, rather than many questionable arguments.
This helps you persuade other people, if you are right, because they won’t see a weaker argument and think all your arguments are that weak.
This helps you be corrected, if you are wrong, because it’s more likely that someone will be able to respond to one or two arguments that you have identified as strong arguments, and show you where you are wrong—no one’s going to do that with 20 weaker arguments, because who has the time?
Put alternately, it also helps with epistemic legibility. It also shows that you aren’t just piling up a lot of soldiers for your side—it shows that you’ve put in the work to weed out the ones which matter, and are not putting weird demands on your reader’s attention by just putting all those which work.
You have a lot of sub-parts in your argument for 1, 2, 3 above. (Like, in the first section there are ~5 points I think are just wrong or misleading, and regardless of whether they are wrong or not are at least highly disputed). It doesn’t help to have succession of such disputed points—regardless of whether your audience is people you agree with or people you don’t!
The way I see the above post (and it’s accompaniment) is knocking down all the soldiers that I’ve encountered talking to lots of people about this over the last few weeks. I would appreciate it if you could stand them back up (because I’m really trying to not be so doomy, and not getting any satisfactory rebuttals).
I think you need to zoom out a bit and look at the implications of these papers. The danger isn’t in what people are doing now, it’s in what they might be doing in a few months following on from this work. The NAS paper was a proof of concept. What happens when it’s massively scaled up? What happens when efficiency gains translate into further efficiency gains?
Probably nothing, honestly.
Here’s a chart of one of the benchmarks the GPT-NAS paper tests on. They GPT-NAS paper is like.… not off trend? Not even SOTA? Honestly looking at all these results my tenative guess is that the differences are basically noise for most techniques; the state space is tiny such that I doubt any of these really leverage actual regularities in it.
From the Abstract:
They weren’t aiming for SOTA! What happens when they do?