If everyone stopped building datacenters you really have made a lot of progress towards stopping the death race (and of course, the algorithm that produces banning datacenter construction would probably not stop there).
I think this is a common misconception. I’m pretty sure algorithmic progress will eventually reach a point where what currently takes a datacenter will be possible on a single machine for a slightly longer training period. If that same algorithm runs on a datacenter it would produce something superhuman, but cutting down to only single-gpu training would then not be enough to completely stop. Algo progress is a slow slog of “grad student descent”, so it likely takes quite a bit longer, and maybe it takes enough longer to figure out alignment. But it doesn’t stop the death race, it just slows it down. Actual stopping would require shredding all silicon big enough to even run the fully trained AI, which doesn’t seem to be in the cards. I’m not saying datacenter construction is good or should continue, or that this won’t buy time, but I think people are wishful-thinking about how much time it buys.
Agree qualitatively (and possible quantitatively). However, there’s a quite large knock-on effect, which is a strong bundle of signals of “AGI is bad, don’t make AGI”. These signals move in various directions between different entities, carrying various messages, but they generally push against AGI. (E.g. signaling legitimacy of the Stop position; the US signaling to other states; society signaling to would-be capabilities researchers; Congress self-signaling “we’re trying to ban this whole thing and will continue to add patches to ban dangerous stuff”; etc.)
I mean, sure, eventually. The key question is how much of algorithmic progress is downstream of hardware scaling. My sense is around 50% of it, maybe a bit more, so that if you cut scaling, progress now happens at around 1/4th of the speed, which is of course huge and makes things a lot better.
Thinking this through step by step in the framework of the AI Futures Model:
First, I’ll check what the model says, then I’ll reconstruct the reasoning behind why it predicts that.
By default, with Daniel’s parameters, Automated Coder (AC) happens in 2030 and ASI happens in 2031 1.33 years later.
If I stop experiment and training compute growth at the start of 2027, then the model predictsAutomated Coder in 2039 rather than 2030. So 4x slower in calendar time (exactly matching habyrka’s guess). It also looks to have well over a 5 year takeoff from AC to ASI as opposed to the default of 1.33 years.
However, this is highly sensitive to the timing of the compute growth pause, because it’s a shock to the flow rather than the stock. e.g. if I instead stop growth at the start of 2029 as in this worksheet, then AC happens in Mar 2031, taking ~2.2 years instead of ~1.2, so slowing things down by <2x. It does still slow down takeoff from AC to ASI to 4 years, so by ~3x (and this is probably at least a slight underestimate because we don’t model hardware R&D automation).
Now I’ll reconstruct why this is the case using simplifications to the model (I actually did these calcluations before plugging the time series things into our model).
Currently, experiment compute is growing at around 3x/year, and human labor around 2x/year. Conditional on no AGI, we’re projecting experiment compute growth to slow to around 2x/year by 2030, and human labor growth to slow to around 1.5x/year.
Figuring out the effect of removing experiment growth is a bit complicated for various reasons.
On the margin, informed by interviews/surveys, we model a ~2.5x gain in “research effort” from 10x more experiment compute. Which if applied instantaneously, would mean a 2.5x slowdown in algorithmic progress.
We estimate that a 100x in human parallel coding labor gives a ~2x in research effort on the margin. We don’t model quantity of research labor, but I expect probably the gains would be relatively small as quality matters a lot more than quantity; let’s shade up to 3x.
So naively based on our model parameters and a simplified version of our model (Cobb-Douglas used to locally approximate a CES), by default the growth in research effort per year from experiment compute is 2.5^log(2-3)=~1.3-1.55x, and from human labor is sqrt(3)^log(1.5-2)=~1.1-1.2x. Meaning that roughly log(1.4)/(log(1.4)+log(1.15))=~70% of research effort growth is coming from experiment compute.
So to simplify from now on, let’s think about what happens if research effort growth has a one-time shock a constant 30% of what it otherwise would have been.
What does this actually mean in terms of the effect on algorithmic/software progess? This means that the shock in research effort growth will eventually cause the software growth rate to be 30% of what it would have been otherwise, but it steadily decreases toward 30% over time (intuitively, the immediate effect is 0 because you have to actually wait for the “missing new experiment compute” to take effect).
The above graph seems to give decent intuition for why the averaged-over-the-relevant-period slowdown in software progress from no more experiment compute might be about 2x (with the 2027.0 growth stoppage), and therefore the overall slowdown from no more compute might be about 4x. It looks like the average of the first 12 years might be close to 0.5.
All of the above is ignoring automation for simplicity. Taking into account automation would mean that more of the gains from labor, so the slowdown is smaller. But on the other hand, as you get more coding labor you get more bottlenecked on experiment compute (which the Cobb-Douglas doesn’t take into account); in the AIFM, you’d eventually get hard bottlenecked, you can have a maximum of 15x research effort gain from coding labor increases alone. Looks like these factors and other deviations from the model might roughly cancel out in the case we’re considering.
But it doesn’t stop the death race, it just slows it down.
My current default assumption is that, yes, someone will build something that obsoletes human intellectual and physical labor. In those futures, the best-case scenario is “Maybe the AIs like keeping humans as pets (and won’t breed us the way we breed pugs).”[1] And the other alternative futures go sharply downhill from there.
So I think of delay in terms of survivor curves, like someone with terminal cancer. How much time can we buy the human race? Can the children alive today get to enjoy a decent lifetime for however long we all have? So I heavily favor delay, in much the same way that I’d favor cancer remission.
A global AI halt might even buy us quite a bit of time.
If I had to bet on a specific model as liking humans and being a responsible “pet owner”, then I currently suspect we might have the best odds with a descendant of Claude. I do actually think that “enculturation” and building morally thoughtful models that like humans gives us non-zero chance of a more acceptable outcome. But I would still prefer humans to control their own destiny.
“Eventually”, sure, but I don’t think that’s operative here. If we had the ASI recipe and could study it safely for ten years, we’d find a way to implement it in a single datacenter. But discovering it in a single data center is much harder. There is actually something missing from current LLMs, there’s a part of intelligence they just don’t have, and the only thing that seems to mitigate that issue is model size, so without ever-increasing model size and analysis of their training dynamics, I think any attempts to get the missing piece are throwing darts with the lights off. (To be fair I have pretty unusual timelines compared to most of LW so maybe what’s convincing to me shouldn’t be to you.)
I agree with the general concern, but it’d be clearly a move in the right direction on that front?
With this kind of proposal I’m more worried that it could lead to a unilateral slowdown just after having animated China to be much more aggressive on AI.
I agree with the general concern, but it’d be clearly a move in the right direction on that front?
agreed, I’m not saying don’t do it. I’m replying to habryka saying it proposes an end to the race.
With this kind of proposal I’m more worried that it could lead to a unilateral slowdown just after having animated China to be much more aggressive on AI.
Doesn’t the USA have enough datacenters to stay ahead of china in the race to fully human-replacing AI for quite a while, even with a unilateral hardware pause? I’m not sure of this claim, but it’s currently my impression that the compute ratio is pretty dramatic.
I expect there’s already enough silicon in place to produce overwhelmingly superhuman AI that still has idiot-savant going on; completely stopping CPU and GPU production worldwide seems to me like it’d be a timeline-extending move, but not by more than a few years. Somewhere between 1.3x and 3x.
Doesn’t the USA have enough datacenters to stay ahead of china in the race to fully human-replacing AI for quite a while, even with a unilateral hardware pause?
As of June 2025 the US has 5x as much compute as China, I’d expect the gap has grown with substantially more American than Chinese data centers coming online in the past ~9 months
If everyone stopped building datacenters you really have made a lot of progress towards stopping the death race (and of course, the algorithm that produces banning datacenter construction would probably not stop there).
I think this is a common misconception. I’m pretty sure algorithmic progress will eventually reach a point where what currently takes a datacenter will be possible on a single machine for a slightly longer training period. If that same algorithm runs on a datacenter it would produce something superhuman, but cutting down to only single-gpu training would then not be enough to completely stop. Algo progress is a slow slog of “grad student descent”, so it likely takes quite a bit longer, and maybe it takes enough longer to figure out alignment. But it doesn’t stop the death race, it just slows it down. Actual stopping would require shredding all silicon big enough to even run the fully trained AI, which doesn’t seem to be in the cards. I’m not saying datacenter construction is good or should continue, or that this won’t buy time, but I think people are wishful-thinking about how much time it buys.
Agree qualitatively (and possible quantitatively). However, there’s a quite large knock-on effect, which is a strong bundle of signals of “AGI is bad, don’t make AGI”. These signals move in various directions between different entities, carrying various messages, but they generally push against AGI. (E.g. signaling legitimacy of the Stop position; the US signaling to other states; society signaling to would-be capabilities researchers; Congress self-signaling “we’re trying to ban this whole thing and will continue to add patches to ban dangerous stuff”; etc.)
I mean, sure, eventually. The key question is how much of algorithmic progress is downstream of hardware scaling. My sense is around 50% of it, maybe a bit more, so that if you cut scaling, progress now happens at around 1/4th of the speed, which is of course huge and makes things a lot better.
Thinking this through step by step in the framework of the AI Futures Model:
First, I’ll check what the model says, then I’ll reconstruct the reasoning behind why it predicts that.
By default, with Daniel’s parameters, Automated Coder (AC) happens in 2030 and ASI happens in 2031 1.33 years later.
If I stop experiment and training compute growth at the start of 2027, then the model predicts Automated Coder in 2039 rather than 2030. So 4x slower in calendar time (exactly matching habyrka’s guess). It also looks to have well over a 5 year takeoff from AC to ASI as opposed to the default of 1.33 years.
I got this by plugging in this modified version of our time series to this unreleased branch of our website.
However, this is highly sensitive to the timing of the compute growth pause, because it’s a shock to the flow rather than the stock. e.g. if I instead stop growth at the start of 2029 as in this worksheet, then AC happens in Mar 2031, taking ~2.2 years instead of ~1.2, so slowing things down by <2x. It does still slow down takeoff from AC to ASI to 4 years, so by ~3x (and this is probably at least a slight underestimate because we don’t model hardware R&D automation).
Now I’ll reconstruct why this is the case using simplifications to the model (I actually did these calcluations before plugging the time series things into our model).
Currently, experiment compute is growing at around 3x/year, and human labor around 2x/year. Conditional on no AGI, we’re projecting experiment compute growth to slow to around 2x/year by 2030, and human labor growth to slow to around 1.5x/year.
Figuring out the effect of removing experiment growth is a bit complicated for various reasons.
On the margin, informed by interviews/surveys, we model a ~2.5x gain in “research effort” from 10x more experiment compute. Which if applied instantaneously, would mean a 2.5x slowdown in algorithmic progress.
We estimate that a 100x in human parallel coding labor gives a ~2x in research effort on the margin. We don’t model quantity of research labor, but I expect probably the gains would be relatively small as quality matters a lot more than quantity; let’s shade up to 3x.
So naively based on our model parameters and a simplified version of our model (Cobb-Douglas used to locally approximate a CES), by default the growth in research effort per year from experiment compute is 2.5^log(2-3)=~1.3-1.55x, and from human labor is sqrt(3)^log(1.5-2)=~1.1-1.2x. Meaning that roughly log(1.4)/(log(1.4)+log(1.15))=~70% of research effort growth is coming from experiment compute.
So to simplify from now on, let’s think about what happens if research effort growth has a one-time shock a constant 30% of what it otherwise would have been.
What does this actually mean in terms of the effect on algorithmic/software progess? This means that the shock in research effort growth will eventually cause the software growth rate to be 30% of what it would have been otherwise, but it steadily decreases toward 30% over time (intuitively, the immediate effect is 0 because you have to actually wait for the “missing new experiment compute” to take effect).
(possibly wrong) I had Claude generate roughly the trajectory of the growth rate change:
The above graph seems to give decent intuition for why the averaged-over-the-relevant-period slowdown in software progress from no more experiment compute might be about 2x (with the 2027.0 growth stoppage), and therefore the overall slowdown from no more compute might be about 4x. It looks like the average of the first 12 years might be close to 0.5.
All of the above is ignoring automation for simplicity. Taking into account automation would mean that more of the gains from labor, so the slowdown is smaller. But on the other hand, as you get more coding labor you get more bottlenecked on experiment compute (which the Cobb-Douglas doesn’t take into account); in the AIFM, you’d eventually get hard bottlenecked, you can have a maximum of 15x research effort gain from coding labor increases alone. Looks like these factors and other deviations from the model might roughly cancel out in the case we’re considering.
My current default assumption is that, yes, someone will build something that obsoletes human intellectual and physical labor. In those futures, the best-case scenario is “Maybe the AIs like keeping humans as pets (and won’t breed us the way we breed pugs).” [1] And the other alternative futures go sharply downhill from there.
So I think of delay in terms of survivor curves, like someone with terminal cancer. How much time can we buy the human race? Can the children alive today get to enjoy a decent lifetime for however long we all have? So I heavily favor delay, in much the same way that I’d favor cancer remission.
A global AI halt might even buy us quite a bit of time.
If I had to bet on a specific model as liking humans and being a responsible “pet owner”, then I currently suspect we might have the best odds with a descendant of Claude. I do actually think that “enculturation” and building morally thoughtful models that like humans gives us non-zero chance of a more acceptable outcome. But I would still prefer humans to control their own destiny.
Do you expect algorithmic progress to never hit diminishing returns?
“Eventually”, sure, but I don’t think that’s operative here. If we had the ASI recipe and could study it safely for ten years, we’d find a way to implement it in a single datacenter. But discovering it in a single data center is much harder. There is actually something missing from current LLMs, there’s a part of intelligence they just don’t have, and the only thing that seems to mitigate that issue is model size, so without ever-increasing model size and analysis of their training dynamics, I think any attempts to get the missing piece are throwing darts with the lights off. (To be fair I have pretty unusual timelines compared to most of LW so maybe what’s convincing to me shouldn’t be to you.)
I agree with the general concern, but it’d be clearly a move in the right direction on that front?
With this kind of proposal I’m more worried that it could lead to a unilateral slowdown just after having animated China to be much more aggressive on AI.
agreed, I’m not saying don’t do it. I’m replying to habryka saying it proposes an end to the race.
Doesn’t the USA have enough datacenters to stay ahead of china in the race to fully human-replacing AI for quite a while, even with a unilateral hardware pause? I’m not sure of this claim, but it’s currently my impression that the compute ratio is pretty dramatic.
I expect there’s already enough silicon in place to produce overwhelmingly superhuman AI that still has idiot-savant going on; completely stopping CPU and GPU production worldwide seems to me like it’d be a timeline-extending move, but not by more than a few years. Somewhere between 1.3x and 3x.
As of June 2025 the US has 5x as much compute as China, I’d expect the gap has grown with substantially more American than Chinese data centers coming online in the past ~9 months
https://epoch.ai/data-insights/ai-supercomputers-performance-share-by-country