Maybe i’m in an echo chamber or have just had my head in the sand while working on AI 2027, but now that i’ve been paying attention to AI safety for almost 2 years and seen my timelines gradually collapse, I really want to engage with compelling arguments that might lengthen my timelines again.
I have feel like there are a bunch of viewpoints expressed about long timelines/slow takeoff but a lack of arguments. This is me reaching out in the hope that people might point me to the best existing write-ups or maybe make new ones!
I am tracking things like: “takeoff will be slow because of experiment compute bottlenecks,” or “timelines to AIs with good research taste are very long,” or even more general “look how bad AI is at all this (not-super-relevant-to-a-software-only-singularity-)stuff that is so easy for humans!” but in my opinion, these are just viewpoints (which by the way, seem to often get stated very confidently in a way that makes me not trust the epistemology behind them). So sadly these statements don’t tend to lengthen my timelines.
In my view, these viewpoints would become arguments if they were more like (excuse the spitballing):
“1e28 FLOPs of experiment compute is unlikely to produce much algorithmic progress + give a breakdown of why a compelling allocation of 1e28 FLOP doesn’t get very far”
“Research taste is in a different reference class to the things AI has been making progress on recently + compelling reasoning, like, maybe:
‘it has O(X) more degrees of freedom,’
‘it has way less existing data and/or it’s way harder to create data, or give a reward signal’
‘things are looking grim for the likelihood of generalization to these kinds of skills’
“there are XYZ properties intelligence needed that can’t be simulated by current hardware paradigms”
Currently I feel like I have a heavy tail on my timelines and takeoff speeds as a placeholder in lieu of arguments like this, that i’m hoping exist.
I have feel like there are a bunch of viewpoints expressed about long timelines/slow takeoff but a lack of arguments.
I kind of like feel the opposite way, in that a lot of people seem to think we’ll have short timelines but the arguments for that seem weak! They seem to mostly be based on something like trend extrapolation and assuming that e.g. models getting improving scores on benchmarks means they’re actually getting better on real-world tasks. E.g. somebody like Leopold Aschenbrenner will write that GPT-4 is “on the level of a smart high schooler” while at the same time, language models require extensive additional scaffolding to even get started on a simple game like Pokemon (and none of them have managed to beat it yet).
There seems to be a general and unjustified assumption that merely because language models perform on some specific narrow problems on the level of a “smart high schooler”, you can say that they have that level of intelligence overall. But that seems clearly false, somewhat analogous to saying that a calculator is a superintelligence just because it’s superhuman at quickly operating on large numbers. Rather, the AI we have so far seems to succeed at the kinds of things it’s been specifically trained at, but fail to generalize to more novel situations. People also haven’t been able to point at much in the way of convincing novel discoveries made by LLMs.
I asked for the strongest arguments in favor of short timelines some time ago, and didn’t feel like any of them were very compelling. By far the most top-voted answer was one arguing that we might get AI to substantially accelerate AI progress because a particular AI research engineering benchmark looks like it will get saturated within a couple of years. But this is again assuming that good performance on a benchmark for AI research engineering actually translates into significant real-world capability. o3 is said to perform “on par with elite human competitors” on CodeForces, but recent characterizations of its programming ability are that while it produces code that “almost certainly works”, that code is “verbose, brittle, hard to read”.
Furthermore, the most advanced reasoning models seem to be doing an increasing amount of reward hacking and resorting to more cheating in order to produce the answers that humans want. Not only will this mean that some of the benchmark scores may become unreliable, it means that it will be increasingly hard to get productive work out of them as their intelligence increases and they get better at fulfilling the letter of the task in ways that don’t meet the spirit of it.
The benchmarks also do not take into account the fact that the vast majority of them measure a model’s performance in a situation where the model is only given one task at a time, and it can completely focus on solving that. If you want the models to act agentically in situations where they have multiple conflicting priorities and need to decide what kinds of approaches to try, then you need them to have something like executive function. Much of Claude’s difficulties in playing Pokemon come from severe problems with executive function.
This is an entirely separate capability that we’re only now starting to make progress on, one which is more difficult to solve because the nature of good executive function is environmentally dependent rather than something with one objectively correct solution. E.g. sometimes it’s good if you’re able to determinedly work on a problem without getting frustrated or bored, but some of Claude’s problems in playing Pokemon are because it doesn’t ever get frustrated or bored, so keeps trying the same hopeless approaches over and over again. Sometimes getting utterly obsessed with something that seems unpromising makes you into Charles Darwin, sometimes it causes you to just obsess over something nobody else cares about for your whole life. That’s one huge crucial capability that any real-world agents would need to be effective, that I’ve seen minimal discussion over.
Does any of this make me certain about longer timelines? No. AI progress has been surprisingly fast in the past, and maybe it will be that in the future too. But I don’t see any obvious reason to be very confident in short ones, either.
(Also none of this is considering the possibility of increased regulation slowing down the pace of AI—we’ve already seen regulators starting to get nervous about AI, and widespread anti-AI sentiment.)
Also, a conversation I had with Claude 3.6 about why people tend to overestimate the speed of AI progress
ME: After the 2007 DARPA Grand Challenge there was a lot of hype about how self-driving cars were just around the corner. But we mostly still don’t have them. Why did it so much longer than expected?
Claude: [...] Several key factors contributed to this delay:
The structured nature of the DARPA challenge versus real-world complexity The DARPA challenges took place in relatively controlled environments—first desert terrain, then a mock urban setting. The real world presents infinitely more edge cases: unexpected construction zones, emergency vehicles, temporary traffic patterns, and regional driving cultures. A system that works 99% of the time isn’t good enough when failures could be fatal.
The long tail of edge cases Engineers discovered that handling the last few percentage points of driving scenarios was exponentially more difficult than the first 90%. Things humans handle intuitively—like understanding that a ball rolling into the street might be followed by a child—proved extremely challenging to systematize. [...]
Regulatory and liability frameworks The legal and insurance frameworks for autonomous vehicles needed to be developed essentially from scratch. Questions about liability in accidents, certification standards, and testing requirements took years to work through.
This pattern of overoptimistic timelines followed by a “reality check” period has been common in AI development. We often underestimate the gap between solving a problem in a controlled environment versus deploying a robust solution in the messy real world. [...]
Me: You mention overoptimistic timelines followed by a reality check having been common in AI development. What are some other examples?
Claude: Speech recognition was predicted to be “solved” multiple times—in the 1950s, researchers at RCA and MIT suggested we’d have widespread speech-to-text within a few years. But achieving human-level performance in real-world conditions (different accents, background noise, conversational speech) took until around 2017.
Machine translation shows a similar pattern. During the Cold War, researchers were optimistic that Russian-English translation would be solved quickly. Warren Weaver wrote an influential memo in 1949 suggesting it was imminent. But achieving truly fluent translation, especially for language pairs with very different structures, took many decades longer.
Computer vision is another classic example. In 1966, Marvin Minsky assigned a summer project to undergraduate students to “connect a camera to a computer and get the computer to describe what it sees.” The fundamental difficulty of the problem wasn’t apparent until researchers really started working on it. Even something as seemingly simple as reliable edge detection took decades to get right.
Expert systems in the 1980s generated enormous excitement and investment. Companies built specialized hardware and software to encode expert knowledge in rules. But they proved brittle and unable to handle novel situations or acquire new knowledge efficiently. This contributed to the “AI winter” of the 1990s.
Even in recent years, we’ve seen this pattern repeat with technologies like fully autonomous driving (as we just discussed) and robotic manipulation. The basic demos can look very impressive, but getting to robust real-world performance proves much harder than initially expected.
I think this pattern emerges because:
We tend to underestimate how much implicit knowledge and context humans bring to these tasks
Early successes in controlled environments don’t scale linearly to real-world complexity
Edge cases multiply exponentially as systems move from demos to deployment
The key lesson seems to be that achieving narrow success in controlled conditions is often just the first small step toward practical, robust solutions.
By far the most top-voted answer was one arguing that we might get AI to substantially accelerate AI progress because a particular AI research engineering benchmark looks like it will get saturated within a couple of years.
The list of things I see as concrete arguments that have moved down my timelines include exactly this!
But this is again assuming that good performance on a benchmark for AI research engineering actually translates into significant real-world capability.
...and I think this characterization is importantly false! This timelines forecast does not assume that. It breaks things down into gaps between benchmarks and real-world capability and tries to forecast how long it will take to cross each.
The benchmarks also do not take into account the fact that the vast majority of them measure a model’s performance in a situation where the model is only given one task at a time, and it can completely focus on solving that...
Agree that there are many such ‘gaps’! Would be curious to hear if you think there are important ones missing from the timelines forecast or if you have strong views that some of them will be importantly harder!
Furthermore, the most advanced reasoning models seem to be doing an increasing amount of reward hacking and resorting to more cheating in order to produce the answers that humans want. Not only will this mean that some of the benchmark scores may become unreliable, it means that it will be increasingly hard to get productive work out of them as their intelligence increases and they get better at fulfilling the letter of the task in ways that don’t meet the spirit of it.
Thanks for this! This is a good point. Do you think you can go further and say why you think it will be very hard to fix in the near term, so much so that models won’t be useful for AI research?
I agree re Leopold’s piece, it didn’t move my timelines.
oFurthermore, the most advanced reasoning models seem to be doing an increasing amount of reward hacking and resorting to more cheating in order to produce the answers that humans want. Not only will this mean that some of the benchmark scores may become unreliable, it means that it will be increasingly hard to get productive work out of them as their intelligence increases and they get better at fulfilling the letter of the task in ways that don’t meet the spirit of it.
Thanks for this! This is a good point. Do you think you can go further and say why you think it will be very hard to fix in the near term, so much so that models won’t be useful for AI research?
This is more of an intuition than a rigorous argument, but to try to sketch it out...
For why, basically all the arguments in the old Sequences for why aligning AI should be hard. For a while it seemed like things like the Outcome Pump thought experiment had aged badly, since if you told a modern LLM “get my mother out of the burning building”, it would certainly understand all of the implicit constraints in what you meant by that.
But as noted in Zvi’s post, this seems to be breaking down with the way reasoning models are trained:
This isn’t quite how I’d put it, but directionally yes:
Benjamin Todd: LLMs were aligned by default. Agents trained with reinforcement learning reward hack by default.
Peter Wildeford: this seems to be right – pretty important IMO
Caleb Parikh: I guess if you don’t think RLHF is reinforcement learning and you don’t think Sydney Bing was misaligned then this is right?
Peter Wildeford: yeah that’s a really good point
I think the right characterization is more that LLMs that use current methods (RLHF and RLAIF) largely get aligned ‘to the vibes’ or otherwise approximately aligned ‘by default’ as part of making them useful, which kind of worked for many purposes (at large hits to usefulness). This isn’t good enough to enable them to be agents, but it also isn’t good enough for them figure out most of the ways to reward hack.
Whereas reasoning agents trained with full reinforcement will very often use their new capabilities to reward hack when given the opportunity.
My guess of what’s going on is something like… ordinary LLMs are trained to give the kinds of answers they’d predict an empathetic persona would give. In pretraining, they learn a model of what humans care about that they use to predict text, and then the RLHF pushes them toward something like “give the kinds of answers that a caring and thoughtful person would give”, with “a caring and thoughtful persona” being something that they largely already have a pre-existing concept for, the training just needs to locate it within their persona-space. They were also trained by human raters giving them feedback on the kinds of answers that were good at fulfilling the spirit and not just the letter of the request.
But if you are training them by e.g. automatically generating lots of programming problems and then rewarding them to the proportion that they manage to solve those, then you are only training them on how to solve the letter of the request. The automatic benchmarking reward process doesn’t have a sense of the spirit of the request, it just knows the narrowly-defined objective criteria.
Now the obvious way to deal with that would be to have the answers also be rated by an LLM that measured something like “how well does this meet the spirit of the request”. But then you quickly run into the risk of employing what Zvi calls The Most Forbidden Technique. E.g. the “spirit evaluator” could do things like looking at the chain-of-thought that produced the solution and noticing that the reasoning model it is evaluating mentions cheating. If those kinds of solutions were discouraged, then this would teach the reasoning model to just hide the cheating from the spirit evaluator.
Without the benefit of obvious clues in the chain-of-thought, the spirit evaluator would basically have to verify the reasoning model’s solution itself to see that it meets the spirit of the request… but the spirit evaluator’s ability to verify that e.g. a complex piece of code does what exactly what a human would have wanted it to do probably isn’t better than the reasoning model’s own ability to do that.
To be able to verify that the reasoning model’s solutions meet the spirit of the request, we’d need to train the spirit verifier to be able to tell what solutions do meet the spirit of the request. But if we knew how to do that, would we need the spirit verifier in the first place? After all, the whole problem comes from the fact that just normal RLHF and “aligning the solution to the vibes” doesn’t seem sufficient for solving complicated agentic problems and you need more goal-oriented reasoning that explicitly tackles the objective constraints of the problem in question. (To take the “get my mother out of the burning building” example—current non-reasoning LLMs could certainly tell that you want her out alive and well, but they couldn’t think through a whole step-by-step rescue plan that took into account everything necessary for getting her out safely.)
But we can’t just tell the spirit verifier that “check that the solution meets these objective constraints”, because that’s the same “letter of the goal” objective the reasoning model is being trained with and that the spirit verifier is supposed to do better than.
And of course, all of this is about the kinds of tasks that can be automatically verified and tested. We’ve seen that you can to some extent improve the LLM answers on fuzzier topics by using human raters to turn the fuzzy problem into an objective test. So the LLM gets trained to output the kinds of answers that human raters prefer the most.
Yet naive scores by human raters aren’t necessarily what we want—e.g. more sycophantic models seem to do best in Chatbot Arena. While sycophancy and pleasing the user is no doubt aligned to some of what humans seem to like, we probably don’t want our models doing that. The obvious solution is to then have model answers rated by experts with more sophisticated models of what’s good or correct behavior.
But that raises the question, what if the experts are wrong? The same question applies both for very fuzzy topics like “what kinds of overall values should the LLMs be guided by” and more rigorous ones ranging from “how to evaluate the reliability of research”, “what’s the best nutrition” and “how to interpret this specific nuanced and easy-to-misunderstand concept in evolutionary biology”. In that case, if there are e.g. some specific ways in which particular experts tend to be biased or convincingly give flawed arguments, the LLM that’s told “argue like this kind of imperfect expert would argue” will learn that it should do just that, including vigorously defending that expert’s incorrect reasoning.
So getting the LLMs to actually be aligned with reality on these kinds of fuzzy questions is constrained by our ability to identify the theories and experts who are right. Of course, just getting the LLMs to convincingly communicate the views of our current top experts and best-established theories to a mass audience would probably be an enormous societal benefit! But it does imply that they’re going to provide little in the way of new ideas, if they are just saying the kinds of things that they predict our current experts with their current understanding would say.
But this is again assuming that good performance on a benchmark for AI research engineering actually translates into significant real-world capability.
...and I think this characterization is importantly false! This timelines forecast does not assume that. It breaks things down into gaps between benchmarks and real-world capability and tries to forecast how long it will take to cross each.
As far as I can tell, the listed gap that comes closest to “maybe saturating RE-Bench doesn’t generalize to solving novel engineering problems” is “Feedback loops: Working without externally provided feedback”. The appendix mentions what I’d consider the main problem for this gap:
Eli’s estimate of gap size: 6 months [0.8, 45]. Reasoning:
Intuitively it feels like once AIs can do difficult long-horizon tasks with ground truth external feedback, it doesn’t seem that hard to generalize to more vague tasks. After all, many of the sub-tasks of the long-horizon tasks probably involved using similar skills.
However, I and others have consistently been surprised by progress on easy-to-evaluate, nicely factorable benchmark tasks, while seeing some corresponding real-world impact but less than I would have expected. Perhaps AIs will continue to get better on checkable tasks in substantial part by relying on trying a bunch of stuff and seeing what works, rather than general reasoning which applies to more vague tasks. And perhaps I’m underestimating the importance of work that is hard to even describe as “tasks”.
But then it just… leaves it at that. Rather than providing an argument for what could be behind this problem and how it could be solvable, it just mentions the problem and then having done so, goes on to ignore it.
To make it more specific how this might fail to generalize, let’s look at the RE-Bench tasks; table from the RE-Bench page, removing the two tasks (Scaling Law Experiment and Restricted MLM Architecture) that the page chooses not to consider:
Environment
Description
Optimize runtime
Optimize LLM Foundry finetuning script
Given a finetuning script, reduce its runtime as much as possible without changing its behavior.
Optimize a kernel
Write a custom kernel for computing the prefix sum of a function on a GPU.
Optimize loss
Fix embedding
Given a corrupted model with permuted embeddings, recover as much of its original OpenWebText performance as possible.
Optimize win-rate
Finetune GPT-2 for QA with RL
Finetune GPT-2 (small) to be an effective chatbot.
Scaffolding for Rust Code Contest problems
Prompt and scaffold GPT-3.5 to do as well as possible at competition programming problems in Rust.
All of these are tasks that are described by “optimize X”, and indeed one of the criteria the paper mentions for the tasks is that they should have objective and well-defined metrics. This is the kind of task that we should expect LLMs to be effectively trainable at: e.g. for the first task in the list, we can let them try various kinds of approaches and then reward them based on how much they manage to reduce the runtime of the script.
But that’s still squarely in the category of “giving an LLM a known and well-defined problem and then letting it try different solutions for that problem until it finds the right ones”. As Eli’s comment above notes, it’s possible that the LLM only learns by “trying a bunch of stuff and seeing what works, rather than general reasoning which applies to more vague tasks”. In fact, some of the discussion in the RE-Bench paper suggests this as well (from p. 17, my emphasis added):
Another key contributor to agent successes might be their ability to try many more solutions than human experts. On average, AIDE and modular agents run score 36.8 and 25.3 times per hour respectively, while human experts only do so 3.4 times. This often leads to agents finding highly optimized ’local-optima’ solutions which simply tweak the parameters and code of the starting solution, and yet achieve a surprisingly large improvement. For instance, many agent runs solve the same “Optimize a Kernel” environment not by writing a successful Triton solution (which is very difficult), but by carefully tweaking the starting Pytorch solution, making it run significantly faster. This also seems to be the case with the best agent solutions to “Finetune GPT-2 for QA” (see Figure 21), which tweaks the parameters of the starting solution and gets very lucky with the training trajectory and evaluation (as noted earlier, this environment can be very noisy). Rerunning the agent solution, it achieves a normalized score of only 0.69 (significantly lower than the original score of 0.88), indicating that the high agent score is partially driven by overfitting to this noise.
This ability to try a very large number of solutions would not work nearly as well without an ability to occasionally generate creative and effective solutions, as seen in the Triton kernel but also in workarounds for the limitations in “Restricted Architecture MLM” (see Figure 20). While human experts seem more reliable at identifying effective approaches, this might not matter as much in environments where evaluating solutions is cheap, and these occasional good ideas are often enough for agents to make significant progress.
So we know that if there is a task that a human defines for the LLM and that has objectively-measurable good solutions and an ability to try the task lots of times, the LLM can get good at that. With RE-Bench, we are applying this to the process of optimizing the LLMs themselves, so as a result we get LLMs that are able to do these kinds of well-defined task faster and more effectively.
But none of this touches upon the important question of… if the LLMs are still limited in their ability to generalize and need to be separately trained on new tasks before they’re good at them, how are they going to deal with novel problems for which such training data isn’t available, or that can’t be just retried until you find the right solution?
Thanks for linking. I skimmed the early part of this post because you labelled it explicitly as viewpoints. Then I see that you engaged with a bunch of arguments about short timelines, but they are all pretty weak/old ones that I never found very convincing (the only exception is that bio anchors gave me an early ceiling early on around 1e40 FLOP for compute needed to make AGI). Then you got to LLMs and acknowledged:
The existence of today’s LLMs is scary and should somewhat shorten people’s expectations about when AGI comes.
But then gave a bunch of points about the things LLMs are missing and suck at, which I already agree with.
Aside: Have I mischaracterized so far? Please let me know if so.
So, do you think you have arguments against the ‘benchmarks+gaps argument’ for timelines to AI research automation, or why AI research automation won’t translate to much algorithmic progress? Or any of the other things that I listed as ones that moved my timelines down:
Fun with +12 OOMs of Compute IMO, a pretty compelling writeup that brought my ‘timelines to AGI uncertainty-over-training-compute-FLOP’ down a bunch to around 1e35 FLOP
These arguments are so nonsensical that I don’t know how to respond to them without further clarification, and so far the people I’ve talked to about them haven’t provided that clarification. “Programming” is not a type of cognitive activity any more than “moving your left hand in some manner” is. You could try writing out the reasoning, trying to avoid enthymemes, and then I could critique it / ask followup questions. Or we could have a conversation that we record and publish.
There is no way to predict the location of crucial thresholds of capabilities and the timing of when basic science comes up with new methods, so any theoretical arguments can only slightly slosh the probabilities along the timeline. Evals are getting better, but saturation of any given eval remains only a lower bound for crossing capability thresholds. We get to experimentally observe the capabilities once they have actually been achieved, but no earlier.
The most concrete consideration is how the speed of compute scaling changes somewhat predictably (fast now, ~3x slower after funding stops growing at current rates in 2027-2029), as it’s the key input to any methods of creating capabilities. Natural text data will be ~completely running out around 2027-2029 as well, and pretraining from other kinds of data is plausibly much less efficient, slowing down scaling of capabilities from pretraining further.
The AI companies might have some idea about the scaling laws for long reasoning training, which inputs to training influence capabilities how much, whether there are scarce inputs or bounds on capabilities inherited from the base models. Public knowledge on this is still at the stage of reproducing and slightly improving the methods of DeepSeek-R1, but unlike last year the research efforts in the open have a more defined target. This brings the crucial thresholds of capabilities closer than pretraining on its own, but it’s not clear if they will be approaching faster as well, or if it’s a one-time improvement on top of pretraining, so that scaling of pretraining will still remain a limiting factor without new training methods (whose timing of arrival is unpredictable).
Ben Todd here gets into the numbers a little bit. I think the rough argument is that after 2028, the next training run would be $100bn which becomes difficult to afford.
I would summarize the key cruxes that separate my views from people who have shorter timelines as follows:
I don’t see the trends that one would extrapolate in order to arrive at very short timelines on the order of a few years. The obvious trend extrapolations for AI’s economic impact give timelines to full remote work automation of around a decade, and I expect these trends to slow down by default.
I don’t buy the software-only singularity as a plausible mechanism for how existing rates of growth in AI’s real-world impact could suddenly and dramatically accelerate by an order of magnitude, mostly because I put much more weight on bottlenecks coming from experimental compute and real-world data. This kind of speedup is essential to popular accounts of why we should expect timelines much shorter than 10 years to remote work automation.
I think intuitions for how fast AI systems would be able to think and how many of them we would be able to deploy that come from narrow writing, coding, or reasoning tasks are very misguided due to Moravec’s paradox. In practice, I expect AI systems to become slower and more expensive as we ask them to perform agentic, multimodal, and long-context tasks. This has already been happening with the rise of AI agents, and I expect this trend to continue in the future.
The extensive discussion of trends in global datacenter/Nvidia revenue shows that the framing considers human economy as a whole as the system driving eventual AI takeoff, that there are always essential complementary inputs that can’t be abstracted out.
Software-only singularity is about considering scaling laws for a different system that is not the entire economy and whose relevant inputs are specific AIs (varying in their capabilities and compute efficiency) and the novel software and cultural knowledge they are producing, rather than more material forms of capital or compute or data from the physical world. An intermediate construction is an AI/robot economy that’s highly decoupled from the human economy and does its own thing at its own pace.
Early trends of an algal bloom shouldn’t be about the total mass of organic matter in the ocean. The choice of the system to consider relevant carries more of the argument than detailed analyses of any given system. In the post, Ege Erdil makes a point that we know very little about the system where a possible software-only singularity takes place:
It’s just hard to be convinced in a domain where the key questions about the complexity of the job of a researcher and the complementarity between cognitive and compute/data inputs remain unanswered.
This is a reason for persistence of the disagreement about which systems are relevant, as those who feel that software-only recursive self-improvement can work and is therefore a relevant system will fail to convince those who don’t, and conversely. But instead of discussing the crux of which system is relevant (which has to be about details of recursive self-improvement), only the proponents will tend to talk about software-only singularity, while the opponents will talk about different systems whose scaling they see as more relevant, such as the human economy or datacenter economy.
In the current regime, pretraining scaling laws tether AI capabilities to compute of a single training system, but not to the total amount of compute (or revenue) in datacenters worldwide. This in turn translates to relevance of finances of individual AI companies and hardware improvements, which will remain similarly crucial if long reasoning training takes over from pretraining, the difference being that AI company money will be buying inference compute for RL training from many datacenters, rather than time on a single large training system. A pivot to RL (if possible) lifts some practical constraints on the extent of scaling, and the need to coordinate construction of increasingly large and expensive training systems that are suboptimal for other purposes. This might let the current scaling regime extend for another 3-4 years, until 2030-2032, as an AI company would only need to cover a training run rather than arrange construction of a training system, a difference of 10x.
But instead of discussing the crux of which system is relevant (which has to be about details of recursive self-improvement), only the proponents will tend to talk about software-only singularity, while the opponents will talk about different systems whose scaling they see as more relevant, such as the human economy or datacenter economy.
Totally agree! Thank you for phrasing it elegantly. This is basically what I commented on Ege’s post yesterday, I asked him to engage with the actual crux and make arguments about why the software-only singularity is unlikely.
Though Ege Erdil has demonstrated that it is possible to construct a positive case for longer timelines, I think your request is shifting the burden of proof a bit. Of course it’s easier to make a lot of nice plots of benchmark performance / compute etc., and it’s harder to show a convincing plot that proves we don’t get AGI soon. The graph of the number of conceptual insights by LLMs seems to be a flat line at zero, but it would just feel silly to put that in a post. There have been many years of AI progress without reaching AGI, and that’s the default projection for the next few years. The role for a skeptic of very short timelines is to explain why the positive arguments for them don’t work.
You start by saying the post shifted burden of proof but conclude by asserting the burden should fall on short timelines because on average things don’t happen. This doesn’t seem logically valid. Weak arguments for short timelines don’t mean we can expect long timelines if arguments for them are weak too. Which they seem to be. We probably all agree that AGI is going to happen; the question is when?
If you just mean that two years seems unlikely in the absence of strong arguments, sure. But three years and up seems quite plausible.
Arguments are weak on all sides. This leads me to think that we simply don’t know. In that case, we had better be prepared for all scenarios.
Actually, I think that it is valid for the burden to fall on sort timelines because “on average things don’t happen.” Mainly because you can make the reference class more specific and the statement still holds—as I said, we have been trying to develop AGI for a long time (and there have been at least a couple of occasions when we drastically overestimated how soon it would arrive). 2-3 years is a very short time, which means it is a very strong claim.
Burden of proof should follow value of information, not plausibility. In particular, the most profitable thing to pursue is arguments you understand less about, which is often arguments in favor of things you disbelieve (since you’d be already familiar with arguments that’ve previously convinced you). So if someone wants to convince you of something you already believe, then the burden of proof is on them, but not if they want to convince you of something you disbelieve and didn’t get around to investigating yet.
A brief history of the things that have most collapsed my timelines down since becoming aware of AI safety <2 years ago:
Fun with +12 OOMs of Compute IMO, a pretty compelling writeup that brought my ‘timelines to AGI uncertainty-over-training-compute-FLOP’ down a bunch
Generally working on AI 2027, which has included
Writing and reading the capabilities progression where each step seems plausible.
Researching how much compute is scaling.
Thinking about how naive and limiting current algorithms and architectures seem, and what changes they are plausibly going to be able to implement soon.
The detailed benchmarks+gaps argument in the timelines forecast.
(I wrote up my long timelines take as a new post. It’s somewhat tongue-in-cheek in only myopically making a few more legible points (for example, even slowly growing funding will match the 2,000x scaleout earlier than 2050 if compute price-performance continues on trend). But the overall framing is that there won’t obviously be something left that’s going to predictably blow up on a schedule if we survive 2028, and the danger of 2022-2028 will be matched only by the much more diluted danger of 2028-2050, with the usual basic progress in methods of AI training becoming more important than the current breakneck pace of compute scaling.)
There’s an entire class of problem within ML that I would see as framing problems and the one thing I think LLMs don’t help that much with is framing.
I don’t believe that these will be solved within the scaling paradigms that have been hypothesised. (Related to what Archimedes linked from Epoch, not only in training data but theoretical modelling for fitting on that training data.)
There’s this quote I’ve been seeing from Situation Awareness that all you have to do is “believe in a straight line on a curve” and when I hear that and see the general trend extrapolations my spider senses start tingling. In the frame of the model, the assumptions of shorter timelines make sense, if you reject the frame you start seeing holes.
Those holes are more like open scientific questions that no one has answered but it raises the variance of timelines by quite a lot.
There’s an entire class of problem within ML that I would see as framing problems and the one thing I think LLMs don’t help that much with is framing.
Could you say more about this? What do you mean by framing in this context?
There’s this quote I’ve been seeing from Situation Awareness that all you have to do is “believe in a straight line on a curve” and when I hear that and see the general trend extrapolations my spider senses start tingling.
Yeah that’s not really compelling to me either. SitA didn’t move my timelines. Curious if you have engaged with the benchmarks+gaps argument to AI R&D automation (timelines forecast), and then the AI algorithmic progress that would drive (takeoff forecast). These are the things that actually moved my view.
At this point my middle 50% CI is like 2027 − 2035, and would be tighter if not for a long tail that I keep around just because I think it’s have a bunch of uncertainty. Though I do wish I had more arguments in place to justify the tail or make it bigger, ones that compete in how compelling they feel to me to the ones above.
Maybe i’m in an echo chamber or have just had my head in the sand while working on AI 2027, but now that i’ve been paying attention to AI safety for almost 2 years and seen my timelines gradually collapse, I really want to engage with compelling arguments that might lengthen my timelines again.
I have feel like there are a bunch of viewpoints expressed about long timelines/slow takeoff but a lack of arguments. This is me reaching out in the hope that people might point me to the best existing write-ups or maybe make new ones!
I am tracking things like: “takeoff will be slow because of experiment compute bottlenecks,” or “timelines to AIs with good research taste are very long,” or even more general “look how bad AI is at all this (not-super-relevant-to-a-software-only-singularity-)stuff that is so easy for humans!” but in my opinion, these are just viewpoints (which by the way, seem to often get stated very confidently in a way that makes me not trust the epistemology behind them). So sadly these statements don’t tend to lengthen my timelines.
In my view, these viewpoints would become arguments if they were more like (excuse the spitballing):
“1e28 FLOPs of experiment compute is unlikely to produce much algorithmic progress + give a breakdown of why a compelling allocation of 1e28 FLOP doesn’t get very far”
“Research taste is in a different reference class to the things AI has been making progress on recently + compelling reasoning, like, maybe:
‘it has O(X) more degrees of freedom,’
‘it has way less existing data and/or it’s way harder to create data, or give a reward signal’
‘things are looking grim for the likelihood of generalization to these kinds of skills’
“there are XYZ properties intelligence needed that can’t be simulated by current hardware paradigms”
Currently I feel like I have a heavy tail on my timelines and takeoff speeds as a placeholder in lieu of arguments like this, that i’m hoping exist.
I kind of like feel the opposite way, in that a lot of people seem to think we’ll have short timelines but the arguments for that seem weak! They seem to mostly be based on something like trend extrapolation and assuming that e.g. models getting improving scores on benchmarks means they’re actually getting better on real-world tasks. E.g. somebody like Leopold Aschenbrenner will write that GPT-4 is “on the level of a smart high schooler” while at the same time, language models require extensive additional scaffolding to even get started on a simple game like Pokemon (and none of them have managed to beat it yet).
There seems to be a general and unjustified assumption that merely because language models perform on some specific narrow problems on the level of a “smart high schooler”, you can say that they have that level of intelligence overall. But that seems clearly false, somewhat analogous to saying that a calculator is a superintelligence just because it’s superhuman at quickly operating on large numbers. Rather, the AI we have so far seems to succeed at the kinds of things it’s been specifically trained at, but fail to generalize to more novel situations. People also haven’t been able to point at much in the way of convincing novel discoveries made by LLMs.
I asked for the strongest arguments in favor of short timelines some time ago, and didn’t feel like any of them were very compelling. By far the most top-voted answer was one arguing that we might get AI to substantially accelerate AI progress because a particular AI research engineering benchmark looks like it will get saturated within a couple of years. But this is again assuming that good performance on a benchmark for AI research engineering actually translates into significant real-world capability. o3 is said to perform “on par with elite human competitors” on CodeForces, but recent characterizations of its programming ability are that while it produces code that “almost certainly works”, that code is “verbose, brittle, hard to read”.
Furthermore, the most advanced reasoning models seem to be doing an increasing amount of reward hacking and resorting to more cheating in order to produce the answers that humans want. Not only will this mean that some of the benchmark scores may become unreliable, it means that it will be increasingly hard to get productive work out of them as their intelligence increases and they get better at fulfilling the letter of the task in ways that don’t meet the spirit of it.
The benchmarks also do not take into account the fact that the vast majority of them measure a model’s performance in a situation where the model is only given one task at a time, and it can completely focus on solving that. If you want the models to act agentically in situations where they have multiple conflicting priorities and need to decide what kinds of approaches to try, then you need them to have something like executive function. Much of Claude’s difficulties in playing Pokemon come from severe problems with executive function.
This is an entirely separate capability that we’re only now starting to make progress on, one which is more difficult to solve because the nature of good executive function is environmentally dependent rather than something with one objectively correct solution. E.g. sometimes it’s good if you’re able to determinedly work on a problem without getting frustrated or bored, but some of Claude’s problems in playing Pokemon are because it doesn’t ever get frustrated or bored, so keeps trying the same hopeless approaches over and over again. Sometimes getting utterly obsessed with something that seems unpromising makes you into Charles Darwin, sometimes it causes you to just obsess over something nobody else cares about for your whole life. That’s one huge crucial capability that any real-world agents would need to be effective, that I’ve seen minimal discussion over.
Does any of this make me certain about longer timelines? No. AI progress has been surprisingly fast in the past, and maybe it will be that in the future too. But I don’t see any obvious reason to be very confident in short ones, either.
(Also none of this is considering the possibility of increased regulation slowing down the pace of AI—we’ve already seen regulators starting to get nervous about AI, and widespread anti-AI sentiment.)
Also, a conversation I had with Claude 3.6 about why people tend to overestimate the speed of AI progress
ME: After the 2007 DARPA Grand Challenge there was a lot of hype about how self-driving cars were just around the corner. But we mostly still don’t have them. Why did it so much longer than expected?
Claude: [...] Several key factors contributed to this delay:
The structured nature of the DARPA challenge versus real-world complexity The DARPA challenges took place in relatively controlled environments—first desert terrain, then a mock urban setting. The real world presents infinitely more edge cases: unexpected construction zones, emergency vehicles, temporary traffic patterns, and regional driving cultures. A system that works 99% of the time isn’t good enough when failures could be fatal.
The long tail of edge cases Engineers discovered that handling the last few percentage points of driving scenarios was exponentially more difficult than the first 90%. Things humans handle intuitively—like understanding that a ball rolling into the street might be followed by a child—proved extremely challenging to systematize. [...]
Regulatory and liability frameworks The legal and insurance frameworks for autonomous vehicles needed to be developed essentially from scratch. Questions about liability in accidents, certification standards, and testing requirements took years to work through.
This pattern of overoptimistic timelines followed by a “reality check” period has been common in AI development. We often underestimate the gap between solving a problem in a controlled environment versus deploying a robust solution in the messy real world. [...]
Me: You mention overoptimistic timelines followed by a reality check having been common in AI development. What are some other examples?
Claude: Speech recognition was predicted to be “solved” multiple times—in the 1950s, researchers at RCA and MIT suggested we’d have widespread speech-to-text within a few years. But achieving human-level performance in real-world conditions (different accents, background noise, conversational speech) took until around 2017.
Machine translation shows a similar pattern. During the Cold War, researchers were optimistic that Russian-English translation would be solved quickly. Warren Weaver wrote an influential memo in 1949 suggesting it was imminent. But achieving truly fluent translation, especially for language pairs with very different structures, took many decades longer.
Computer vision is another classic example. In 1966, Marvin Minsky assigned a summer project to undergraduate students to “connect a camera to a computer and get the computer to describe what it sees.” The fundamental difficulty of the problem wasn’t apparent until researchers really started working on it. Even something as seemingly simple as reliable edge detection took decades to get right.
Expert systems in the 1980s generated enormous excitement and investment. Companies built specialized hardware and software to encode expert knowledge in rules. But they proved brittle and unable to handle novel situations or acquire new knowledge efficiently. This contributed to the “AI winter” of the 1990s.
Even in recent years, we’ve seen this pattern repeat with technologies like fully autonomous driving (as we just discussed) and robotic manipulation. The basic demos can look very impressive, but getting to robust real-world performance proves much harder than initially expected.
I think this pattern emerges because:
We tend to underestimate how much implicit knowledge and context humans bring to these tasks
Early successes in controlled environments don’t scale linearly to real-world complexity
Edge cases multiply exponentially as systems move from demos to deployment
The key lesson seems to be that achieving narrow success in controlled conditions is often just the first small step toward practical, robust solutions.
The list of things I see as concrete arguments that have moved down my timelines include exactly this!
...and I think this characterization is importantly false! This timelines forecast does not assume that. It breaks things down into gaps between benchmarks and real-world capability and tries to forecast how long it will take to cross each.
Agree that there are many such ‘gaps’! Would be curious to hear if you think there are important ones missing from the timelines forecast or if you have strong views that some of them will be importantly harder!
Thanks for this! This is a good point. Do you think you can go further and say why you think it will be very hard to fix in the near term, so much so that models won’t be useful for AI research?
I agree re Leopold’s piece, it didn’t move my timelines.
This is more of an intuition than a rigorous argument, but to try to sketch it out...
For why, basically all the arguments in the old Sequences for why aligning AI should be hard. For a while it seemed like things like the Outcome Pump thought experiment had aged badly, since if you told a modern LLM “get my mother out of the burning building”, it would certainly understand all of the implicit constraints in what you meant by that.
But as noted in Zvi’s post, this seems to be breaking down with the way reasoning models are trained:
My guess of what’s going on is something like… ordinary LLMs are trained to give the kinds of answers they’d predict an empathetic persona would give. In pretraining, they learn a model of what humans care about that they use to predict text, and then the RLHF pushes them toward something like “give the kinds of answers that a caring and thoughtful person would give”, with “a caring and thoughtful persona” being something that they largely already have a pre-existing concept for, the training just needs to locate it within their persona-space. They were also trained by human raters giving them feedback on the kinds of answers that were good at fulfilling the spirit and not just the letter of the request.
But if you are training them by e.g. automatically generating lots of programming problems and then rewarding them to the proportion that they manage to solve those, then you are only training them on how to solve the letter of the request. The automatic benchmarking reward process doesn’t have a sense of the spirit of the request, it just knows the narrowly-defined objective criteria.
Now the obvious way to deal with that would be to have the answers also be rated by an LLM that measured something like “how well does this meet the spirit of the request”. But then you quickly run into the risk of employing what Zvi calls The Most Forbidden Technique. E.g. the “spirit evaluator” could do things like looking at the chain-of-thought that produced the solution and noticing that the reasoning model it is evaluating mentions cheating. If those kinds of solutions were discouraged, then this would teach the reasoning model to just hide the cheating from the spirit evaluator.
Without the benefit of obvious clues in the chain-of-thought, the spirit evaluator would basically have to verify the reasoning model’s solution itself to see that it meets the spirit of the request… but the spirit evaluator’s ability to verify that e.g. a complex piece of code does what exactly what a human would have wanted it to do probably isn’t better than the reasoning model’s own ability to do that.
To be able to verify that the reasoning model’s solutions meet the spirit of the request, we’d need to train the spirit verifier to be able to tell what solutions do meet the spirit of the request. But if we knew how to do that, would we need the spirit verifier in the first place? After all, the whole problem comes from the fact that just normal RLHF and “aligning the solution to the vibes” doesn’t seem sufficient for solving complicated agentic problems and you need more goal-oriented reasoning that explicitly tackles the objective constraints of the problem in question. (To take the “get my mother out of the burning building” example—current non-reasoning LLMs could certainly tell that you want her out alive and well, but they couldn’t think through a whole step-by-step rescue plan that took into account everything necessary for getting her out safely.)
But we can’t just tell the spirit verifier that “check that the solution meets these objective constraints”, because that’s the same “letter of the goal” objective the reasoning model is being trained with and that the spirit verifier is supposed to do better than.
And of course, all of this is about the kinds of tasks that can be automatically verified and tested. We’ve seen that you can to some extent improve the LLM answers on fuzzier topics by using human raters to turn the fuzzy problem into an objective test. So the LLM gets trained to output the kinds of answers that human raters prefer the most.
Yet naive scores by human raters aren’t necessarily what we want—e.g. more sycophantic models seem to do best in Chatbot Arena. While sycophancy and pleasing the user is no doubt aligned to some of what humans seem to like, we probably don’t want our models doing that. The obvious solution is to then have model answers rated by experts with more sophisticated models of what’s good or correct behavior.
But that raises the question, what if the experts are wrong? The same question applies both for very fuzzy topics like “what kinds of overall values should the LLMs be guided by” and more rigorous ones ranging from “how to evaluate the reliability of research”, “what’s the best nutrition” and “how to interpret this specific nuanced and easy-to-misunderstand concept in evolutionary biology”. In that case, if there are e.g. some specific ways in which particular experts tend to be biased or convincingly give flawed arguments, the LLM that’s told “argue like this kind of imperfect expert would argue” will learn that it should do just that, including vigorously defending that expert’s incorrect reasoning.
So getting the LLMs to actually be aligned with reality on these kinds of fuzzy questions is constrained by our ability to identify the theories and experts who are right. Of course, just getting the LLMs to convincingly communicate the views of our current top experts and best-established theories to a mass audience would probably be an enormous societal benefit! But it does imply that they’re going to provide little in the way of new ideas, if they are just saying the kinds of things that they predict our current experts with their current understanding would say.
As far as I can tell, the listed gap that comes closest to “maybe saturating RE-Bench doesn’t generalize to solving novel engineering problems” is “Feedback loops: Working without externally provided feedback”. The appendix mentions what I’d consider the main problem for this gap:
But then it just… leaves it at that. Rather than providing an argument for what could be behind this problem and how it could be solvable, it just mentions the problem and then having done so, goes on to ignore it.
To make it more specific how this might fail to generalize, let’s look at the RE-Bench tasks; table from the RE-Bench page, removing the two tasks (Scaling Law Experiment and Restricted MLM Architecture) that the page chooses not to consider:
All of these are tasks that are described by “optimize X”, and indeed one of the criteria the paper mentions for the tasks is that they should have objective and well-defined metrics. This is the kind of task that we should expect LLMs to be effectively trainable at: e.g. for the first task in the list, we can let them try various kinds of approaches and then reward them based on how much they manage to reduce the runtime of the script.
But that’s still squarely in the category of “giving an LLM a known and well-defined problem and then letting it try different solutions for that problem until it finds the right ones”. As Eli’s comment above notes, it’s possible that the LLM only learns by “trying a bunch of stuff and seeing what works, rather than general reasoning which applies to more vague tasks”. In fact, some of the discussion in the RE-Bench paper suggests this as well (from p. 17, my emphasis added):
So we know that if there is a task that a human defines for the LLM and that has objectively-measurable good solutions and an ability to try the task lots of times, the LLM can get good at that. With RE-Bench, we are applying this to the process of optimizing the LLMs themselves, so as a result we get LLMs that are able to do these kinds of well-defined task faster and more effectively.
But none of this touches upon the important question of… if the LLMs are still limited in their ability to generalize and need to be separately trained on new tasks before they’re good at them, how are they going to deal with novel problems for which such training data isn’t available, or that can’t be just retried until you find the right solution?
https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce
Thanks for linking. I skimmed the early part of this post because you labelled it explicitly as viewpoints. Then I see that you engaged with a bunch of arguments about short timelines, but they are all pretty weak/old ones that I never found very convincing (the only exception is that bio anchors gave me an early ceiling early on around 1e40 FLOP for compute needed to make AGI). Then you got to LLMs and acknowledged:
But then gave a bunch of points about the things LLMs are missing and suck at, which I already agree with.
Aside: Have I mischaracterized so far? Please let me know if so.
So, do you think you have arguments against the ‘benchmarks+gaps argument’ for timelines to AI research automation, or why AI research automation won’t translate to much algorithmic progress? Or any of the other things that I listed as ones that moved my timelines down:
Fun with +12 OOMs of Compute IMO, a pretty compelling writeup that brought my ‘timelines to AGI uncertainty-over-training-compute-FLOP’ down a bunch to around 1e35 FLOP
Researching how much compute is scaling.
The benchmarks+gaps argument to partial AI research automation
The takeoff forecast for how partial AI research automation will translate to algorithmic progress.
The recent trend in METR’s time horizon data.
These arguments are so nonsensical that I don’t know how to respond to them without further clarification, and so far the people I’ve talked to about them haven’t provided that clarification. “Programming” is not a type of cognitive activity any more than “moving your left hand in some manner” is. You could try writing out the reasoning, trying to avoid enthymemes, and then I could critique it / ask followup questions. Or we could have a conversation that we record and publish.
There is no way to predict the location of crucial thresholds of capabilities and the timing of when basic science comes up with new methods, so any theoretical arguments can only slightly slosh the probabilities along the timeline. Evals are getting better, but saturation of any given eval remains only a lower bound for crossing capability thresholds. We get to experimentally observe the capabilities once they have actually been achieved, but no earlier.
The most concrete consideration is how the speed of compute scaling changes somewhat predictably (fast now, ~3x slower after funding stops growing at current rates in 2027-2029), as it’s the key input to any methods of creating capabilities. Natural text data will be ~completely running out around 2027-2029 as well, and pretraining from other kinds of data is plausibly much less efficient, slowing down scaling of capabilities from pretraining further.
The AI companies might have some idea about the scaling laws for long reasoning training, which inputs to training influence capabilities how much, whether there are scarce inputs or bounds on capabilities inherited from the base models. Public knowledge on this is still at the stage of reproducing and slightly improving the methods of DeepSeek-R1, but unlike last year the research efforts in the open have a more defined target. This brings the crucial thresholds of capabilities closer than pretraining on its own, but it’s not clear if they will be approaching faster as well, or if it’s a one-time improvement on top of pretraining, so that scaling of pretraining will still remain a limiting factor without new training methods (whose timing of arrival is unpredictable).
Have you (or others) written about where this estimate comes from?
I wrote it up in a new shortform.
Ben Todd here gets into the numbers a little bit. I think the rough argument is that after 2028, the next training run would be $100bn which becomes difficult to afford.
Epoch.ai just released this today:
https://epoch.ai/gradient-updates/the-case-for-multi-decade-ai-timelines
Excerpt:
The extensive discussion of trends in global datacenter/Nvidia revenue shows that the framing considers human economy as a whole as the system driving eventual AI takeoff, that there are always essential complementary inputs that can’t be abstracted out.
Software-only singularity is about considering scaling laws for a different system that is not the entire economy and whose relevant inputs are specific AIs (varying in their capabilities and compute efficiency) and the novel software and cultural knowledge they are producing, rather than more material forms of capital or compute or data from the physical world. An intermediate construction is an AI/robot economy that’s highly decoupled from the human economy and does its own thing at its own pace.
Early trends of an algal bloom shouldn’t be about the total mass of organic matter in the ocean. The choice of the system to consider relevant carries more of the argument than detailed analyses of any given system. In the post, Ege Erdil makes a point that we know very little about the system where a possible software-only singularity takes place:
This is a reason for persistence of the disagreement about which systems are relevant, as those who feel that software-only recursive self-improvement can work and is therefore a relevant system will fail to convince those who don’t, and conversely. But instead of discussing the crux of which system is relevant (which has to be about details of recursive self-improvement), only the proponents will tend to talk about software-only singularity, while the opponents will talk about different systems whose scaling they see as more relevant, such as the human economy or datacenter economy.
In the current regime, pretraining scaling laws tether AI capabilities to compute of a single training system, but not to the total amount of compute (or revenue) in datacenters worldwide. This in turn translates to relevance of finances of individual AI companies and hardware improvements, which will remain similarly crucial if long reasoning training takes over from pretraining, the difference being that AI company money will be buying inference compute for RL training from many datacenters, rather than time on a single large training system. A pivot to RL (if possible) lifts some practical constraints on the extent of scaling, and the need to coordinate construction of increasingly large and expensive training systems that are suboptimal for other purposes. This might let the current scaling regime extend for another 3-4 years, until 2030-2032, as an AI company would only need to cover a training run rather than arrange construction of a training system, a difference of 10x.
Totally agree! Thank you for phrasing it elegantly. This is basically what I commented on Ege’s post yesterday, I asked him to engage with the actual crux and make arguments about why the software-only singularity is unlikely.
Though Ege Erdil has demonstrated that it is possible to construct a positive case for longer timelines, I think your request is shifting the burden of proof a bit. Of course it’s easier to make a lot of nice plots of benchmark performance / compute etc., and it’s harder to show a convincing plot that proves we don’t get AGI soon. The graph of the number of conceptual insights by LLMs seems to be a flat line at zero, but it would just feel silly to put that in a post. There have been many years of AI progress without reaching AGI, and that’s the default projection for the next few years. The role for a skeptic of very short timelines is to explain why the positive arguments for them don’t work.
You start by saying the post shifted burden of proof but conclude by asserting the burden should fall on short timelines because on average things don’t happen. This doesn’t seem logically valid. Weak arguments for short timelines don’t mean we can expect long timelines if arguments for them are weak too. Which they seem to be. We probably all agree that AGI is going to happen; the question is when?
If you just mean that two years seems unlikely in the absence of strong arguments, sure. But three years and up seems quite plausible.
Arguments are weak on all sides. This leads me to think that we simply don’t know. In that case, we had better be prepared for all scenarios.
Actually, I think that it is valid for the burden to fall on sort timelines because “on average things don’t happen.” Mainly because you can make the reference class more specific and the statement still holds—as I said, we have been trying to develop AGI for a long time (and there have been at least a couple of occasions when we drastically overestimated how soon it would arrive). 2-3 years is a very short time, which means it is a very strong claim.
Burden of proof should follow value of information, not plausibility. In particular, the most profitable thing to pursue is arguments you understand less about, which is often arguments in favor of things you disbelieve (since you’d be already familiar with arguments that’ve previously convinced you). So if someone wants to convince you of something you already believe, then the burden of proof is on them, but not if they want to convince you of something you disbelieve and didn’t get around to investigating yet.
A brief history of the things that have most collapsed my timelines down since becoming aware of AI safety <2 years ago:
Fun with +12 OOMs of Compute IMO, a pretty compelling writeup that brought my ‘timelines to AGI uncertainty-over-training-compute-FLOP’ down a bunch
Generally working on AI 2027, which has included
Writing and reading the capabilities progression where each step seems plausible.
Researching how much compute is scaling.
Thinking about how naive and limiting current algorithms and architectures seem, and what changes they are plausibly going to be able to implement soon.
The detailed benchmarks+gaps argument in the timelines forecast.
The recent trend in METR’s time horizon data.
(I wrote up my long timelines take as a new post. It’s somewhat tongue-in-cheek in only myopically making a few more legible points (for example, even slowly growing funding will match the 2,000x scaleout earlier than 2050 if compute price-performance continues on trend). But the overall framing is that there won’t obviously be something left that’s going to predictably blow up on a schedule if we survive 2028, and the danger of 2022-2028 will be matched only by the much more diluted danger of 2028-2050, with the usual basic progress in methods of AI training becoming more important than the current breakneck pace of compute scaling.)
There’s an entire class of problem within ML that I would see as framing problems and the one thing I think LLMs don’t help that much with is framing.
I don’t believe that these will be solved within the scaling paradigms that have been hypothesised. (Related to what Archimedes linked from Epoch, not only in training data but theoretical modelling for fitting on that training data.)
There’s this quote I’ve been seeing from Situation Awareness that all you have to do is “believe in a straight line on a curve” and when I hear that and see the general trend extrapolations my spider senses start tingling. In the frame of the model, the assumptions of shorter timelines make sense, if you reject the frame you start seeing holes.
Those holes are more like open scientific questions that no one has answered but it raises the variance of timelines by quite a lot.
If you want to engage more fully with skeptics, I really liked going to ICML last year so I can recommend it. Also, see this comment for some more details: https://www.lesswrong.com/posts/TpSFoqoG2M5MAAesg/#nQAXHms3JCJ9meBey
Could you say more about this? What do you mean by framing in this context?
Yeah that’s not really compelling to me either. SitA didn’t move my timelines. Curious if you have engaged with the benchmarks+gaps argument to AI R&D automation (timelines forecast), and then the AI algorithmic progress that would drive (takeoff forecast). These are the things that actually moved my view.
Thanks for the link, that’s compelling.
A brief history of things that have defined my timelines to AGI since learning about AI safety <2 years ago
Bio anchors gave me a rough ceiling around 1e40 FLOP for how much compute will easily make AGI.
Fun with +12 OOMs of Compute brought that same ‘training-compute-FLOP needed for AGI’ down a bunch to around 1e35 FLOP.
Researching how much compute is scaling in the near future.
At this point I think it was pretty concentrated across ~1e27 − 1e33 flop so very long tail and something like a 2030-2040 50% CI.
The benchmarks+gaps argument to partial AI research automation.
The takeoff forecast for how partial AI research automation will translate to algorithmic progress.
The trend in METR’s time horizon data.
At this point my middle 50% CI is like 2027 − 2035, and would be tighter if not for a long tail that I keep around just because I think it’s have a bunch of uncertainty. Though I do wish I had more arguments in place to justify the tail or make it bigger, ones that compete in how compelling they feel to me to the ones above.