An important fact that influences many of my predictions about AI timelines and the capability of AI systems in the near term (even conditional on a pause) is that we have really no way of upper-bounding the capability of today’s AIs given reasonable elicitation.
Take the statement “today’s best AIs could be used to automate 95% of current AI R&D tasks, given 10% as much compute as was used to pretrain them and a strong team working for 4 years”. I think most people in the AI xrisk community, even those who expect transformative AI in the next few years, think that statement is false. But as far as I can tell, we have no way to falsify it.
One way to falsify such a hypothesis is to spend a ton of effort eliciting a particular model on various tasks using many different methods. We basically don’t do this these days for at least a couple reasons.
First, progress in the field is so fast that models are obsoleted quickly. We probably spend a little under a year per base model[1] and only a few months per flagship post-train model;[2] it doesn’t make tons of sense to keep working with a particular model—trying to fine-tune or scaffold/prompt it—once there are much better/cheaper ones.
Second, we basically have new tasks every couple of years or so (e.g., 2023-2024 was chatbots, 2025-2026 is largely focused on agentic coding). Our task suite changes every couple years (or less), so the target of what we’re eliciting for does as well.
There are also some reasons why elicitation is just hard.
Sometimes I encounter people who, when I say “elicitation”, think “scaling inference compute”. But that’s only a small part of it. A practical definition of elicitation (one that would answer the question I posed above) would include not just different methods of using more inference time compute, but also scaffolding, prompting, fine-tuning (SFT, RL, maybe more), activation steering, and new methods for all of these. Remember, we didn’t have “reasoning models” publicly until late 2024, there’s probably much more out there.
The last time humanity fully elicited an AI model was probably 2018 (and the following few years) with the BERT family. Before that, we saw a huge amount of use of ResNet-50 (2015). I think that these models were well elicited because thousands of researchers spent years working with them and trying to get them to solve different tasks.
Due to limited elicitation, we can’t confidently upper-bound how useful current AIs would be for real world tasks if, for example, there was a pause on new AI training. I think it’s plausible that current AIs could automate almost all of current AI research and a large fraction of cognitive labor, given the right elicitation (e.g., a few serial years, thousands of independent efforts, no new pretraining). This has many implications.
I argued here that companies should switch base models very frequently due to it being cost-effective to do so. Chinese AI companies who publish more information seem to train new flagship base models every 8 months on average (ChatGPT).
AI is progressing so rapidly that the field doesn’t have time to do high ROI elicitation because there are even higher ROI AI R&D pathways present in pretraining and post-training. It’s analogous to how we didn’t invent multi-core CPUs until Dennard scaling stopped, or we didn’t use multi-patterning until lithography wavelength stalled.
The abundance of independent moderate-to-high ROI paths to better chips is why Moore’s Law continued for so long, and for the same reason, it seems unlikely that AI R&D will naturally hit a wall, and nontrivial to enforce a pause.
I don’t see how “nontrivial to enforce a pause” follows (beyond it requiring some amount of international coordination at some point which is unrelated to your point). It was my impression that the continued progress in e.g. Moore’s law requires a comparable steady increase in investment and R&D spending, so if the government does a large scale intervention on the inputs, this could effectively halt progress in many ways that matter. (Some forms of algorithmic progress seem indeed hard to halt without aggressive enforcement)
Mostly I meant that government needs to block all the source of progress—pretraining, posttraining, data, elicitation, others—rather than just some if it wants to mostly or entirely halt progress.
Moore’s law has required steady increase in investment, but investment has increased much less than the rate of Moore’s law, which is about 1.5x/year. If R&D spending were constant, progress might slow down to quadratic or cubic rather than exponential, still very fast.
The situation with AI is even tougher because if a software-only singularity is possible, growth will by definition continue to be exponential or faster if we halt growth in all inputs other than algorithmic efficiency, just a slower exponential than it would have been if investment continued ramping up. If status quo is 6 months from software singularity, than constant compute would mean perhaps 1 year to the singularity, and to delay the singularity until 10 years, governments must reduce inputs by at least 10x.
Take the statement “today’s best AIs could be used to automate 95% of current AI R&D tasks, given 10% as much compute as was used to pretrain them and a strong team working for 4 years”. I think most people in the AI xrisk community, even those who expect transformative AI in the next few years, think that statement is false. But as far as I can tell, we have no way to falsify it.
I don’t know what “automate 95% of current AI R&D tasks” really means and depending on the definition I think this is maybe already true without any further elicitation required, but assuming you mean “nearly fully automate AI R&D”, I think it would be possible to get a pretty good sense with effort using trend extrapolation and effort. It wouldn’t be easy, but I think we could become more confident and I think we already have some understanding based on a bunch of less direct evidence.
An important fact that influences many of my predictions about AI timelines and the capability of AI systems in the near term (even conditional on a pause) is that we have really no way of upper-bounding the capability of today’s AIs given reasonable elicitation.
Take the statement “today’s best AIs could be used to automate 95% of current AI R&D tasks, given 10% as much compute as was used to pretrain them and a strong team working for 4 years”. I think most people in the AI xrisk community, even those who expect transformative AI in the next few years, think that statement is false. But as far as I can tell, we have no way to falsify it.
One way to falsify such a hypothesis is to spend a ton of effort eliciting a particular model on various tasks using many different methods. We basically don’t do this these days for at least a couple reasons.
First, progress in the field is so fast that models are obsoleted quickly. We probably spend a little under a year per base model[1] and only a few months per flagship post-train model;[2] it doesn’t make tons of sense to keep working with a particular model—trying to fine-tune or scaffold/prompt it—once there are much better/cheaper ones.
Second, we basically have new tasks every couple of years or so (e.g., 2023-2024 was chatbots, 2025-2026 is largely focused on agentic coding). Our task suite changes every couple years (or less), so the target of what we’re eliciting for does as well.
There are also some reasons why elicitation is just hard.
Sometimes I encounter people who, when I say “elicitation”, think “scaling inference compute”. But that’s only a small part of it. A practical definition of elicitation (one that would answer the question I posed above) would include not just different methods of using more inference time compute, but also scaffolding, prompting, fine-tuning (SFT, RL, maybe more), activation steering, and new methods for all of these. Remember, we didn’t have “reasoning models” publicly until late 2024, there’s probably much more out there.
The last time humanity fully elicited an AI model was probably 2018 (and the following few years) with the BERT family. Before that, we saw a huge amount of use of ResNet-50 (2015). I think that these models were well elicited because thousands of researchers spent years working with them and trying to get them to solve different tasks.
Due to limited elicitation, we can’t confidently upper-bound how useful current AIs would be for real world tasks if, for example, there was a pause on new AI training. I think it’s plausible that current AIs could automate almost all of current AI research and a large fraction of cognitive labor, given the right elicitation (e.g., a few serial years, thousands of independent efforts, no new pretraining). This has many implications.
I argued here that companies should switch base models very frequently due to it being cost-effective to do so. Chinese AI companies who publish more information seem to train new flagship base models every 8 months on average (ChatGPT).
For instance, OpenAI seems to release a new best model every 3.5 months or so (ChatGPT).
AI is progressing so rapidly that the field doesn’t have time to do high ROI elicitation because there are even higher ROI AI R&D pathways present in pretraining and post-training. It’s analogous to how we didn’t invent multi-core CPUs until Dennard scaling stopped, or we didn’t use multi-patterning until lithography wavelength stalled.
The abundance of independent moderate-to-high ROI paths to better chips is why Moore’s Law continued for so long, and for the same reason, it seems unlikely that AI R&D will naturally hit a wall, and nontrivial to enforce a pause.
I don’t see how “nontrivial to enforce a pause” follows (beyond it requiring some amount of international coordination at some point which is unrelated to your point).
It was my impression that the continued progress in e.g. Moore’s law requires a comparable steady increase in investment and R&D spending, so if the government does a large scale intervention on the inputs, this could effectively halt progress in many ways that matter. (Some forms of algorithmic progress seem indeed hard to halt without aggressive enforcement)
Mostly I meant that government needs to block all the source of progress—pretraining, posttraining, data, elicitation, others—rather than just some if it wants to mostly or entirely halt progress.
Moore’s law has required steady increase in investment, but investment has increased much less than the rate of Moore’s law, which is about 1.5x/year. If R&D spending were constant, progress might slow down to quadratic or cubic rather than exponential, still very fast.
The situation with AI is even tougher because if a software-only singularity is possible, growth will by definition continue to be exponential or faster if we halt growth in all inputs other than algorithmic efficiency, just a slower exponential than it would have been if investment continued ramping up. If status quo is 6 months from software singularity, than constant compute would mean perhaps 1 year to the singularity, and to delay the singularity until 10 years, governments must reduce inputs by at least 10x.
I don’t know what “automate 95% of current AI R&D tasks” really means and depending on the definition I think this is maybe already true without any further elicitation required, but assuming you mean “nearly fully automate AI R&D”, I think it would be possible to get a pretty good sense with effort using trend extrapolation and effort. It wouldn’t be easy, but I think we could become more confident and I think we already have some understanding based on a bunch of less direct evidence.