Why I find AI 2027 unconvincing as a reason to expect very short (1-3 years) timelines
(Originally a Facebook comment)
To me, the developments of the last few years look like “LLMs keep getting better at the thing they’re good at (learning to do the kinds of things they’re explicitly trained to do and where it’s easy to provide good training data), while still breaking relatively easily if you go sufficiently off-distribution, with LLMs making no progress toward transcending their fundamental limits. We’re just throwing more and more training data on them so there’s a larger amount of everyday tasks that becomes in-distribution, but anything like a FOOM would require e.g. the kind of genuine causal reasoning ability that LLMs have consistently failed to demonstrate”.
So basically still the kinds of things that I described in my post “Surprising LLM reasoning failures” a year ago. In that post, I wrote
So if continued model scaling had continued to bring benefits, I might have expected that these issues would eventually get fixed through just that. But the fact that e.g. GPT-4.5 was disappointing and that larger models have generally not brought expected improvements, implies that further scaling won’t significantly help with generalization. (This is a crux for me: if scaling got back on track on producing improved generalization at the rate it used to, then I would update toward this possibly being able to take us to AGI.)
What’s the current status of that crux? Well, it’s complicated. On one hand, progress clearly ended up not stalling with GPT-4.5 after all, and models have continued to get better at generalizing when they have been fed more training data. On the other, the Claude Mythos Preview system card lists, on page 38-41, examples that look like the kind of old “from everything else the system can do, you’d really expect it to be able to not screw these ones up, but it does” LLM failure mode. (Excerpts 1 and 2; excerpt 3 looks like it might be explainable by reward hacking rather than being the more classic kind of LLM failure mode.)
The system card then goes on to say:
These have been known issues with our models in general, with multiple FTEs dedicated to improving on them, for months; we do not believe that they would be generally resolved via more persistent feedback, better elicitation, etc.
So, it still looks to me like some kind of qualitative breakthrough would be required to make LLMs capable of doing things that are sufficiently out-of-distribution relative to their training data. Also they still hallucinate and so on.
On my read, AI 2027 seems to deal with this problem by… pretending that the issue doesn’t exist. It does very little to consider the various limitations of current LLMs and to discuss reasons why we should expect them to be overcome in the near future. I found the site pretty frustrating to try to read because of this and haven’t spent much time looking at it, but Claude also didn’t find any explicit discussion about this when I asked it:
What’s missing or treated only glancingly [at AI 2027]:
* Causal reasoning, counterfactual reasoning, Pearl’s hierarchy. Not discussed.
* Compositional generalization and systematic out-of-distribution behavior. Not discussed.
* Symbol grounding and the relationship between text-trained representations and world structure. Not discussed.
* Data efficiency as a marker of qualitative differences in learning. Mentioned only obliquely (the 4,000x figure), and treated as a quantity that automated researchers will grind down rather than as evidence of a possibly different learning regime. [...]
The structural reason for this gap is that the methodology drives it. The timelines forecast extrapolates from METR’s measured time horizon trend; the takeoff forecast extrapolates from researcher surveys about R&D progress multipliers. Both methodologies treat capability as a single dimension that’s getting longer/faster/cheaper, and both work by reasoning about how quickly that dimension extends. There’s no place in the formal model where “LLMs might have a qualitative limitation that doesn’t show up on time-horizon benchmarks until suddenly it does” can be represented. The model can express uncertainty about rate, but not really about whether the thing being measured is the right thing to extrapolate.
To the extent that AI 2027 says something about new architectures, it handwaves a bit in the direction of IDA and neuralese, but on my (and Claude’s) read, neither of these obviously addresses the current limitations of LLMs. Hallucinations are briefly mentioned as something that is an issue for Agent-1 and then OpenBrain is later said to reassure the president that they are no longer an issue for Agent-3, but there is no argument for how this would happen.
So while I don’t doubt that AI 2027′s predictions might be correct for as long as we are extrapolating the kinds of things LLMs are good at, it seems to me meaningless past that.
Now of course, there are a lot of potential things that one could try and that a lot of people are trying, that could turn out to be the qualitative breakthrough we need to get to an AI that could do transcend the limitations of current LLMs. If we were talking about, say, a 10-year horizon—then yeah, the possibility of ASI within that timeframe would sound a lot more plausible to me.
But going from “this novel architecture outperforms LLMs in a toy setup” to “this is actually a superintelligence” isn’t something that would happen in just a year or two (not even with LLMs writing all your code for you). You need to figure out how to train it (both in terms of capability and alignment—even if you were a total disbeliever in risks from AGI, some minimum amount of alignment is still necessary if you want it to do anything for you), actually do those training runs, redesign and manufacture any new custom chips that this architecture might use, deal with any other issues that crop up, etc..
As one comparison, the first paper to show the potential for something like “training a language model to produce intermediate steps of reasoning before outputting its final answer” was arguably “Show Your Work: Scratchpads for Intermediate Computation with Language Models” (November 2021). The first commercially-ready reasoning model was OpenAI’s o1, out in December 2024 (or in September of the same year, if you count o1-preview).
So even for a relatively minor tweak that was closer to “let’s train our systems differently” than “let’s figure out an entirely new architecture or combination of them”, it took three years to go from the first paper proposing it to a commercially useful product. I would expect any drastic revision to take longer than that.
Separately from all that, I think there’s also various other issues that e.g. “Current AIs seem pretty misaligned to me” is pointing at (it frames things in terms of misalignment, but clearly those are also capability problems) around designing an AI’s motivations. Many of the classical arguments for alignment difficulty also end up becoming arguments for capability difficulty (an AI with a propensity to reward hack is likely to have that thwart its attempt to take over the world).
And I think there are some open problems around motivation/goal prioritization that haven’t even been discussed much and would need to be solved for something to become an open-ended superintelligence. (While not being a major problem for current-day LLMs that just do what they’re told and let their human users figure out prioritization issues, or that only do prioritization inside narrow domains like code where objectively correct answer exist and can be trained.) Where the nature of the problems might be such that you need to do something like trial-and-error in the real world to solve them (with a large fraction of the AIs trying it failing), rather than them just dropping out of general capability like AI 2027-type models assume.
Why I find AI 2027 unconvincing as a reason to expect very short (1-3 years) timelines
(Originally a Facebook comment)
To me, the developments of the last few years look like “LLMs keep getting better at the thing they’re good at (learning to do the kinds of things they’re explicitly trained to do and where it’s easy to provide good training data), while still breaking relatively easily if you go sufficiently off-distribution, with LLMs making no progress toward transcending their fundamental limits. We’re just throwing more and more training data on them so there’s a larger amount of everyday tasks that becomes in-distribution, but anything like a FOOM would require e.g. the kind of genuine causal reasoning ability that LLMs have consistently failed to demonstrate”.
So basically still the kinds of things that I described in my post “Surprising LLM reasoning failures” a year ago. In that post, I wrote
What’s the current status of that crux? Well, it’s complicated. On one hand, progress clearly ended up not stalling with GPT-4.5 after all, and models have continued to get better at generalizing when they have been fed more training data. On the other, the Claude Mythos Preview system card lists, on page 38-41, examples that look like the kind of old “from everything else the system can do, you’d really expect it to be able to not screw these ones up, but it does” LLM failure mode. (Excerpts 1 and 2; excerpt 3 looks like it might be explainable by reward hacking rather than being the more classic kind of LLM failure mode.)
The system card then goes on to say:
So, it still looks to me like some kind of qualitative breakthrough would be required to make LLMs capable of doing things that are sufficiently out-of-distribution relative to their training data. Also they still hallucinate and so on.
On my read, AI 2027 seems to deal with this problem by… pretending that the issue doesn’t exist. It does very little to consider the various limitations of current LLMs and to discuss reasons why we should expect them to be overcome in the near future. I found the site pretty frustrating to try to read because of this and haven’t spent much time looking at it, but Claude also didn’t find any explicit discussion about this when I asked it:
What’s missing or treated only glancingly [at AI 2027]:
* Causal reasoning, counterfactual reasoning, Pearl’s hierarchy. Not discussed.
* Compositional generalization and systematic out-of-distribution behavior. Not discussed.
* Symbol grounding and the relationship between text-trained representations and world structure. Not discussed.
* Data efficiency as a marker of qualitative differences in learning. Mentioned only obliquely (the 4,000x figure), and treated as a quantity that automated researchers will grind down rather than as evidence of a possibly different learning regime. [...]
The structural reason for this gap is that the methodology drives it. The timelines forecast extrapolates from METR’s measured time horizon trend; the takeoff forecast extrapolates from researcher surveys about R&D progress multipliers. Both methodologies treat capability as a single dimension that’s getting longer/faster/cheaper, and both work by reasoning about how quickly that dimension extends. There’s no place in the formal model where “LLMs might have a qualitative limitation that doesn’t show up on time-horizon benchmarks until suddenly it does” can be represented. The model can express uncertainty about rate, but not really about whether the thing being measured is the right thing to extrapolate.
To the extent that AI 2027 says something about new architectures, it handwaves a bit in the direction of IDA and neuralese, but on my (and Claude’s) read, neither of these obviously addresses the current limitations of LLMs. Hallucinations are briefly mentioned as something that is an issue for Agent-1 and then OpenBrain is later said to reassure the president that they are no longer an issue for Agent-3, but there is no argument for how this would happen.
So while I don’t doubt that AI 2027′s predictions might be correct for as long as we are extrapolating the kinds of things LLMs are good at, it seems to me meaningless past that.
Now of course, there are a lot of potential things that one could try and that a lot of people are trying, that could turn out to be the qualitative breakthrough we need to get to an AI that could do transcend the limitations of current LLMs. If we were talking about, say, a 10-year horizon—then yeah, the possibility of ASI within that timeframe would sound a lot more plausible to me.
But going from “this novel architecture outperforms LLMs in a toy setup” to “this is actually a superintelligence” isn’t something that would happen in just a year or two (not even with LLMs writing all your code for you). You need to figure out how to train it (both in terms of capability and alignment—even if you were a total disbeliever in risks from AGI, some minimum amount of alignment is still necessary if you want it to do anything for you), actually do those training runs, redesign and manufacture any new custom chips that this architecture might use, deal with any other issues that crop up, etc..
As one comparison, the first paper to show the potential for something like “training a language model to produce intermediate steps of reasoning before outputting its final answer” was arguably “Show Your Work: Scratchpads for Intermediate Computation with Language Models” (November 2021). The first commercially-ready reasoning model was OpenAI’s o1, out in December 2024 (or in September of the same year, if you count o1-preview).
So even for a relatively minor tweak that was closer to “let’s train our systems differently” than “let’s figure out an entirely new architecture or combination of them”, it took three years to go from the first paper proposing it to a commercially useful product. I would expect any drastic revision to take longer than that.
Separately from all that, I think there’s also various other issues that e.g. “Current AIs seem pretty misaligned to me” is pointing at (it frames things in terms of misalignment, but clearly those are also capability problems) around designing an AI’s motivations. Many of the classical arguments for alignment difficulty also end up becoming arguments for capability difficulty (an AI with a propensity to reward hack is likely to have that thwart its attempt to take over the world).
And I think there are some open problems around motivation/goal prioritization that haven’t even been discussed much and would need to be solved for something to become an open-ended superintelligence. (While not being a major problem for current-day LLMs that just do what they’re told and let their human users figure out prioritization issues, or that only do prioritization inside narrow domains like code where objectively correct answer exist and can be trained.) Where the nature of the problems might be such that you need to do something like trial-and-error in the real world to solve them (with a large fraction of the AIs trying it failing), rather than them just dropping out of general capability like AI 2027-type models assume.