i’ve become more skeptical of rsi over time. here’s my current best guess at what happens as we automate ai research.
for the next several years, ai will provide a bigger and bigger efficiency multiplier to the workflow of a human ai researcher.
ai assistants will probably not uniformly make researchers faster across the board, but rather make certain kinds of things way faster and other kinds of things only a little bit faster.
in fact probably it will make some things 100x faster, a lot of things 2x faster, and then be literally useless for a lot of remaining things
amdahl’s law tells us that we will mostly be bottlenecked on the things that don’t get sped up a ton. like if the thing that got sped up 100x was only 10% of the original thing, then you don’t get more than a 1/(1 − 10%) speedup.
i think the speedup is a bit more than amdahl’s law implies. task X took up 10% of the time because there is diminishing returns to doing more X, and so you’d ideally do exactly the amount of X such that the marginal value of time spent on X is exactly in equilibrium with time spent on anything else. if you suddenly decrease the cost of X substantially, the equilibrium point shifts towards doing more X.
in other words, if AI makes lit review really cheap, you probably want to do a much more thorough lit review than you otherwise would have, rather than just doing the same amount of lit review but cheaper.
at the first moment that ai can fully replace a human researcher (that is, you can purely just put more compute in and get more research out, and only negligible human labor is required), the ai will probably be more expensive per unit of research than the human
(things get a little bit weird because my guess is before ai can drop-in replace a human, we will reach a point where adding ai assistance equivalent to the cost of 100 humans to 2025-era openai research would be equally as good as adding 100 humans, but the ai’s are not doing the same things as the humans, and if you just keep adding ai’s you start experiencing diminishing returns faster than with adding humans. i think my analysis still mostly holds despite this)
naively, this means that the first moment that AIs can fully automate AI research at human-cost is not a special criticality threshold. if you are at equilibrium for allocating money between researchers and compute, then suddenly having the ability to convert compute into researchers at the exchange rate of the salary of a human researcher doesn’t really make sense
in reality, you will probably not be at equilibrium, because there are a lot of inefficiencies in hiring humans—recruiting is a lemon market, you have to onboard new hires relatively slowly, management capacity is limited, there is a inelastic and inefficient supply of qualified hires, etc. but i claim this is a relatively small effect and can’t explain a one OOM increase in workforce size
also: anyone who has worked in a large organization knows that team size is not everything. having too many people can often even be a liability and slow you down. even when it doesn’t, adding more people almost never makes your team linearly more productive.
however, if AIs have much better scaling laws with additional parallel compute than human organizations do, then this could change things a lot. this is one of my biggest uncertainties here and one reason i still take rsi seriously.
your AIs might higher have bandwidth communication with each other than your humans do. but also maybe they might be worse at generalizing previous findings to new situations or something.
they might be more aligned with doing lots of research all day, whereas humans care about a lot of other things like money and status and fun and so on. but if outer alignment is hard we might get the AI equivalent of corporate politics.
one other thing is that compute is a necessary input to research. i’ll mostly roll this into the compute cost of actually running the AIs.
the part where AI research feeds back into how good the AIs are could be very slow in practice
there are logarithmic returns to more pretraining compute and more test time compute. so an improvement that 10xes the effective compute doesn’t actually get you that much. 4.5 isn’t that much better than 4 despite being 10x more compute (which is in turn not that much better than 3.5, I would claim).
you run out of low hanging fruit at some point. each 2x in compute efficiency is harder to find than the previous one.
i would claim that in fact much of the recent feeling that AI progress is fast is due to a lot of low hanging fruit being picked. for example, the shift from pretrained models to RL for reasoning picked a lot of low hanging fruit due to not using test time compute / not eliciting CoTs well, and we shouldn’t expect the same kind of jump consistently.
an emotional angle: exponentials can feel very slow in practice; for example, moore’s law is kind of insane when you think about it (doubling every 18 months is pretty fast), but it still takes decades to play out
My current best guess median is that we’ll see 6 OOMs of effective compute in the first year after full automation of AI R&D if this occurs in ~2029 using a 1e29 training run and compute is scaled up by a factor of 3.5x[1] over the course of this year[2]. This is around 5 years of progress at the current rate[3].
How big of a deal is 6 OOMs? I think it’s a pretty big deal; I have a draft post discussing how much an OOM gets you (on top of full automation of AI R&D) that I should put out somewhat soon.
Further, my distribution over this is radically uncertain with a 25th percentile of 2.5 OOMs (2 years of progress) and a 75th percentile of 12 OOMs.
The short breakdown of the key claims is:
Initial progress will be fast, perhaps ~15x faster algorithmic progress than humans.
Progress will probably speed up before slowing down due to training smarter AIs that can accelerate progress even faster, and this being faster than returns diminish on software.
We’ll be quite far from the limits of software progress (perhaps median 12 OOMs) at the point when we first achieve full automation.
Here is a somewhat summarized and rough version of the argument (stealing heavily from some of Tom Davidson’s forthcoming work):
At the point of full automation, progress will be fast:
Probably you’ll have lots of parallel workers running pretty fast at the point when you have full automation or shortly after this. This isn’t totally obvious due to inference compute, but prices often drop fast.
My guess is you’ll have enough compute that if you use 1⁄6 of your compute running AIs, you’ll be able to run the equivalent of ~1 million AIs which are roughly as good as the best human research scientists+engineers (taking into account cost reductions for using weaker models for many tasks). This is attempting to account for a reduction in the number of models due to using a bunch of inference compute. You’ll be able to run these AIs at the equivalent of 60x speed (3x from hours, 5x from direct speed, 2x from coordination, and 2x from variable time compute and/or context swapping with a cheaper+faster model). So, like 15k parallel copies at 60x speed.
Probably the AI company has like ~3k researchers, but when you adjust for quality, this is only as good as like 600 of the top engineers/researchers.
Let’s say marginal returns to parallelism are roughly 0.55.
Then, the increase in “serial labor equivalents” is roughly (15k / 600)^0.55 * 60 = 350. (Note that most of this is from speed and quality rather than parallel copies!)
Production of algorithmic research is due to both compute and labor. So, let’s say Cobb-Douglas with labor^0.5 compute^0.5 (My guess is that current marginal returns are more like labor^0.6 compute^0.4, but returns will get worse as you add more labor.) So, we do 350^0.5 = 19x which roughly matches my 15x speed up median.
I expect progress to slow as you hit limits or run out of progress doable with cheap experiments, but limits are probably pretty high as discussed in the next bullet.
Limits are high:
Human brain is 1e24 training flop.
We’re using 1e29 flop to get a bit over human level, so like 4 OOMs of headroom from this.
We can probably get a lot more efficiency, maybe 9 OOMs, at least for efficiency up rather than down. (By efficiency up rather than down I mean: we can do as well as using 1e33 flop with 1e24 real flop relative to scaling at the point when you hit human efficiency, but probably can’t train a human-level AI for 1e15 real flop.)
At some point, I’ll write a post that makes a better version of this argument and presents a full version of my picture.
I don’t think we’ll see a speed criticality per se; rather, I expect the rate of progress to accelerate up to the point of full automation. But I currently don’t think this makes a huge difference to the bottom line of “progress in the first year after full automation in practice”, as I expect to initially see fast cost decreases and inference time compute can only go so far. I could expand this argument to the extent you have cruxes like “slower takeoff because we’ve already eaten low-hanging fruit with earlier AI acceleration” and “inference compute means you hit full automation much faster”.
That is, after the first company fully automates AI R&D internally, if they decide to go as fast as possible and their AIs/employees/others don’t try to sabotage these efforts. And I’m assuming that AI software progress hasn’t substantially slowed down by the time of full automation, though conditioning on a 1e29 training run means that at least compute scaling progress (which is a key driver of software progress) hasn’t slowed down all that much.
This seems to make the simplifying assumption that the R&D automation is applied to a large fraction of all the compute that was previously driving algorithmic progress right?
If we imagine that a company only owns 10% of the compute being used to drive algorithmic progress pre-automation (and is only responsible for say 30% of its own algorithmic progress, with the rest coming from other labs/academia/open-source), and this company is the only one automating their AI R&D, then the effect on overall progress might be reduced (the 15X multiplier only applies to 30% of the relevant algorithmic progress).
In practice I would guess that either the leading actor has enough of a lead that they are already responsible for most of their algorithmic progress, or other groups are close behind and will thus automate their own AI R&D around the same time anyway. But I could imagine this slowing down the impact of initial AI R&D automation a little bit (and it might make a big difference for questions like “how much would it accelerate a non-frontier lab that stole the model weights and tried to do rsi”).
Yes, I think frontier AI companies are responsible for most of the algorithmic progress. I think its unclear how much the leading actor benefits from progress done at other slightly behind AI companies and this could make progress substantially slower. (However, it’s possible the leading AI company would be able to acquire the GPUs from these other companies.)
at the first moment that ai can fully replace a human researcher (that is, you can purely just put more compute in and get more research out, and only negligible human labor is required), the ai will probably be more expensive per unit of research than the human
Why do you think this? It seems to me that for most tasks once an AI gets some skill it is much cheaper to run it for that skill than a human.
timelines takes
i’ve become more skeptical of rsi over time. here’s my current best guess at what happens as we automate ai research.
for the next several years, ai will provide a bigger and bigger efficiency multiplier to the workflow of a human ai researcher.
ai assistants will probably not uniformly make researchers faster across the board, but rather make certain kinds of things way faster and other kinds of things only a little bit faster.
in fact probably it will make some things 100x faster, a lot of things 2x faster, and then be literally useless for a lot of remaining things
amdahl’s law tells us that we will mostly be bottlenecked on the things that don’t get sped up a ton. like if the thing that got sped up 100x was only 10% of the original thing, then you don’t get more than a 1/(1 − 10%) speedup.
i think the speedup is a bit more than amdahl’s law implies. task X took up 10% of the time because there is diminishing returns to doing more X, and so you’d ideally do exactly the amount of X such that the marginal value of time spent on X is exactly in equilibrium with time spent on anything else. if you suddenly decrease the cost of X substantially, the equilibrium point shifts towards doing more X.
in other words, if AI makes lit review really cheap, you probably want to do a much more thorough lit review than you otherwise would have, rather than just doing the same amount of lit review but cheaper.
at the first moment that ai can fully replace a human researcher (that is, you can purely just put more compute in and get more research out, and only negligible human labor is required), the ai will probably be more expensive per unit of research than the human
(things get a little bit weird because my guess is before ai can drop-in replace a human, we will reach a point where adding ai assistance equivalent to the cost of 100 humans to 2025-era openai research would be equally as good as adding 100 humans, but the ai’s are not doing the same things as the humans, and if you just keep adding ai’s you start experiencing diminishing returns faster than with adding humans. i think my analysis still mostly holds despite this)
naively, this means that the first moment that AIs can fully automate AI research at human-cost is not a special criticality threshold. if you are at equilibrium for allocating money between researchers and compute, then suddenly having the ability to convert compute into researchers at the exchange rate of the salary of a human researcher doesn’t really make sense
in reality, you will probably not be at equilibrium, because there are a lot of inefficiencies in hiring humans—recruiting is a lemon market, you have to onboard new hires relatively slowly, management capacity is limited, there is a inelastic and inefficient supply of qualified hires, etc. but i claim this is a relatively small effect and can’t explain a one OOM increase in workforce size
also: anyone who has worked in a large organization knows that team size is not everything. having too many people can often even be a liability and slow you down. even when it doesn’t, adding more people almost never makes your team linearly more productive.
however, if AIs have much better scaling laws with additional parallel compute than human organizations do, then this could change things a lot. this is one of my biggest uncertainties here and one reason i still take rsi seriously.
your AIs might higher have bandwidth communication with each other than your humans do. but also maybe they might be worse at generalizing previous findings to new situations or something.
they might be more aligned with doing lots of research all day, whereas humans care about a lot of other things like money and status and fun and so on. but if outer alignment is hard we might get the AI equivalent of corporate politics.
one other thing is that compute is a necessary input to research. i’ll mostly roll this into the compute cost of actually running the AIs.
the part where AI research feeds back into how good the AIs are could be very slow in practice
there are logarithmic returns to more pretraining compute and more test time compute. so an improvement that 10xes the effective compute doesn’t actually get you that much. 4.5 isn’t that much better than 4 despite being 10x more compute (which is in turn not that much better than 3.5, I would claim).
you run out of low hanging fruit at some point. each 2x in compute efficiency is harder to find than the previous one.
i would claim that in fact much of the recent feeling that AI progress is fast is due to a lot of low hanging fruit being picked. for example, the shift from pretrained models to RL for reasoning picked a lot of low hanging fruit due to not using test time compute / not eliciting CoTs well, and we shouldn’t expect the same kind of jump consistently.
an emotional angle: exponentials can feel very slow in practice; for example, moore’s law is kind of insane when you think about it (doubling every 18 months is pretty fast), but it still takes decades to play out
My current best guess median is that we’ll see 6 OOMs of effective compute in the first year after full automation of AI R&D if this occurs in ~2029 using a 1e29 training run and compute is scaled up by a factor of 3.5x[1] over the course of this year[2]. This is around 5 years of progress at the current rate[3].
How big of a deal is 6 OOMs? I think it’s a pretty big deal; I have a draft post discussing how much an OOM gets you (on top of full automation of AI R&D) that I should put out somewhat soon.
Further, my distribution over this is radically uncertain with a 25th percentile of 2.5 OOMs (2 years of progress) and a 75th percentile of 12 OOMs.
The short breakdown of the key claims is:
Initial progress will be fast, perhaps ~15x faster algorithmic progress than humans.
Progress will probably speed up before slowing down due to training smarter AIs that can accelerate progress even faster, and this being faster than returns diminish on software.
We’ll be quite far from the limits of software progress (perhaps median 12 OOMs) at the point when we first achieve full automation.
Here is a somewhat summarized and rough version of the argument (stealing heavily from some of Tom Davidson’s forthcoming work):
At the point of full automation, progress will be fast:
Probably you’ll have lots of parallel workers running pretty fast at the point when you have full automation or shortly after this. This isn’t totally obvious due to inference compute, but prices often drop fast.
My guess is you’ll have enough compute that if you use 1⁄6 of your compute running AIs, you’ll be able to run the equivalent of ~1 million AIs which are roughly as good as the best human research scientists+engineers (taking into account cost reductions for using weaker models for many tasks). This is attempting to account for a reduction in the number of models due to using a bunch of inference compute. You’ll be able to run these AIs at the equivalent of 60x speed (3x from hours, 5x from direct speed, 2x from coordination, and 2x from variable time compute and/or context swapping with a cheaper+faster model). So, like 15k parallel copies at 60x speed.
Probably the AI company has like ~3k researchers, but when you adjust for quality, this is only as good as like 600 of the top engineers/researchers.
Let’s say marginal returns to parallelism are roughly 0.55.
Then, the increase in “serial labor equivalents” is roughly (15k / 600)^0.55 * 60 = 350. (Note that most of this is from speed and quality rather than parallel copies!)
Production of algorithmic research is due to both compute and labor. So, let’s say Cobb-Douglas with labor^0.5 compute^0.5 (My guess is that current marginal returns are more like labor^0.6 compute^0.4, but returns will get worse as you add more labor.) So, we do 350^0.5 = 19x which roughly matches my 15x speed up median.
Progress will speed up before slowing down:
See discussion here
I expect progress to slow as you hit limits or run out of progress doable with cheap experiments, but limits are probably pretty high as discussed in the next bullet.
Limits are high:
Human brain is 1e24 training flop.
We’re using 1e29 flop to get a bit over human level, so like 4 OOMs of headroom from this.
We can probably get a lot more efficiency, maybe 9 OOMs, at least for efficiency up rather than down. (By efficiency up rather than down I mean: we can do as well as using 1e33 flop with 1e24 real flop relative to scaling at the point when you hit human efficiency, but probably can’t train a human-level AI for 1e15 real flop.)
At some point, I’ll write a post that makes a better version of this argument and presents a full version of my picture.
I don’t think we’ll see a speed criticality per se; rather, I expect the rate of progress to accelerate up to the point of full automation. But I currently don’t think this makes a huge difference to the bottom line of “progress in the first year after full automation in practice”, as I expect to initially see fast cost decreases and inference time compute can only go so far. I could expand this argument to the extent you have cruxes like “slower takeoff because we’ve already eaten low-hanging fruit with earlier AI acceleration” and “inference compute means you hit full automation much faster”.
3.5x is roughly the rate of bare metal compute scale-up per year.
That is, after the first company fully automates AI R&D internally, if they decide to go as fast as possible and their AIs/employees/others don’t try to sabotage these efforts. And I’m assuming that AI software progress hasn’t substantially slowed down by the time of full automation, though conditioning on a 1e29 training run means that at least compute scaling progress (which is a key driver of software progress) hasn’t slowed down all that much.
I think we see about 1.2 OOMs per year including both hardware and software.
Thanks for these thoughtful predictions. Do you think there’s anything we can do today to prepare for accelerated or automated AI research?
Maybe distracting technicality:
This seems to make the simplifying assumption that the R&D automation is applied to a large fraction of all the compute that was previously driving algorithmic progress right?
If we imagine that a company only owns 10% of the compute being used to drive algorithmic progress pre-automation (and is only responsible for say 30% of its own algorithmic progress, with the rest coming from other labs/academia/open-source), and this company is the only one automating their AI R&D, then the effect on overall progress might be reduced (the 15X multiplier only applies to 30% of the relevant algorithmic progress).
In practice I would guess that either the leading actor has enough of a lead that they are already responsible for most of their algorithmic progress, or other groups are close behind and will thus automate their own AI R&D around the same time anyway. But I could imagine this slowing down the impact of initial AI R&D automation a little bit (and it might make a big difference for questions like “how much would it accelerate a non-frontier lab that stole the model weights and tried to do rsi”).
Yes, I think frontier AI companies are responsible for most of the algorithmic progress. I think its unclear how much the leading actor benefits from progress done at other slightly behind AI companies and this could make progress substantially slower. (However, it’s possible the leading AI company would be able to acquire the GPUs from these other companies.)
Why do you think this? It seems to me that for most tasks once an AI gets some skill it is much cheaper to run it for that skill than a human.