In this case, they’d need the capital stock to self-replicate once per month. ($1tr * 2^9 = $512tr.) I’d bet against physical capital self-replicating this quickly immediately after AGI – here I estimate that after AGI physical capital will have a ~one year doubling time. But it might be possible, for example if AGI can make very rapid technological progress.
Why look at the number immediately after AGI, here, rather than further into the intelligence explosion? (Where we expect growth speeds to get much faster.) Or earlier in the intelligence explosion, for that matter.
(Also, more minor point: Even if we start the clock at AGI, it’s fine if the initial doubling speed is just ~2.4 months, if the rate of production continuously increase so as to double every 3 doublings.)
Why look at the number immediately after AGI, here, rather than further into the intelligence explosion?
Here was the thought:
If the lab hasn’t outgrown RoW by the time RoW develops AGI then we’re now in a position where RoW has more physical capital than the lab. This means that, in the next physical doubling, RoW will produce more physical capital than the lab, and so do more learning by doing than the lab has ever done. After that doubling, RoW will then have better tech than the lab and so their physical capital will double more quickly. So, after that doubling, RoW will have both more physical capital and their physical capital will double more quickly. This means they’re on a trajectory to always have more physical capital and the lab can’t outgrow them.
So this is implicitly assuming that the number of units produced is the hard bottleneck on the technology of physical capital, and the lab’s additional AI cognitive labour doesn’t help. This is in line with the classic economist view that you bump up against ultimate limits to what can be inferred from data. But I think qualitatively smarter AI could change this a lot. Maybe the lab has much smarter AI than RoW and so can learn much more per unit produced.
So, thinking about it more, I think this calc makes sense if we start the clock at the point when the SIE (software-only intelligence explosion) fizzles out. Because then, once RoW catches up, they’ll have ~equal cognitive inputs to the lab. So the lab really does need to overtake RoW on physical capital before this point. (Maybe the lab can get more chips than RoW? But this is a type of physical capital.)
Anyway, accounting for this makes it more plausible the lab can outgrow RoW. They need the 1 month doubling time robots not when they first have AGI, but when the software intelligence explosion has fizzled. Which means they have more powerful AI and longer to do robot experiments.
Another reason my calc understates the lab’s chance is that there may be higher value from experiments conducted inserial. RoW constructs all their physical capital in parallel in one doubling. Whereas the lab does it over multiple doublings one after the other. That could make a big difference.
How to do better?
The more nuanced way to game this out would be to represent, at each time-step, the physical capital, cognitive labour, and technology (for physical capital) of both the lab and RoW.
Then have a production function for how inputs of physical capital produced and cognitive labour improve technology in each time-step. Probably smg like g_A = (K^a * C^b)^lambda * A^-beta. Where a+b=1, lambda represents the benefits of doing research in serial, and beta accounts for ideas getting harder to find. (This accounts for your other point about the rate of production doubling every 3 doublings. The parameter beta controls whether it’s “3” or some other number.)
Then in each timestep simulate each actor’s change in:
Physical capital
the level of technology determines the physical capital doubling time
so you combine current physical capital and technology to get new physical capital
Technology
Apply the equation above for g_A.
Cognitive labour.
Probably simplest to make this exogenous. Specify some trajectory given your views on the software-only intelligence explosion.
My overall view
I think that physical capital doubling times of a few months shortly after AGI are more plausible than when I first wrote that appendix. So I think this is plausible if we’re starting the clock at the end of the SIE.
Accounting improving serial experiments will favour the lab further. As will the possibility the lab can maintain a cognitive advantage beyond the SIE by designing and quickly manufacturing amazing chips.
But the assumption that a company can spend $1trillion on physical capital before the end of the SIE still seems too high. Though it’s unrealistic to think RoW will use all their physical capital for growth, which balances this out.
The assumption of a 9 month lead feels generous.
Maybe the most plausible scenarios involve ~6 month leads, ~$250b on physical capital, ~1 month robot doubling times at the end of the SIE, and doubling times becoming faster over time.
I made a simple of the dynamic here. (Still not modelling the gains from serial experiments and
So overall I think I’d still bet against the lab being in a position to do this on the economic fundamentals; but it is more plausible that I’d thought.
Thanks for this!
Why look at the number immediately after AGI, here, rather than further into the intelligence explosion? (Where we expect growth speeds to get much faster.) Or earlier in the intelligence explosion, for that matter.
(Also, more minor point: Even if we start the clock at AGI, it’s fine if the initial doubling speed is just ~2.4 months, if the rate of production continuously increase so as to double every 3 doublings.)
Here was the thought:
If the lab hasn’t outgrown RoW by the time RoW develops AGI then we’re now in a position where RoW has more physical capital than the lab. This means that, in the next physical doubling, RoW will produce more physical capital than the lab, and so do more learning by doing than the lab has ever done. After that doubling, RoW will then have better tech than the lab and so their physical capital will double more quickly. So, after that doubling, RoW will have both more physical capital and their physical capital will double more quickly. This means they’re on a trajectory to always have more physical capital and the lab can’t outgrow them.
So this is implicitly assuming that the number of units produced is the hard bottleneck on the technology of physical capital, and the lab’s additional AI cognitive labour doesn’t help. This is in line with the classic economist view that you bump up against ultimate limits to what can be inferred from data. But I think qualitatively smarter AI could change this a lot. Maybe the lab has much smarter AI than RoW and so can learn much more per unit produced.
So, thinking about it more, I think this calc makes sense if we start the clock at the point when the SIE (software-only intelligence explosion) fizzles out. Because then, once RoW catches up, they’ll have ~equal cognitive inputs to the lab. So the lab really does need to overtake RoW on physical capital before this point. (Maybe the lab can get more chips than RoW? But this is a type of physical capital.)
Anyway, accounting for this makes it more plausible the lab can outgrow RoW. They need the 1 month doubling time robots not when they first have AGI, but when the software intelligence explosion has fizzled. Which means they have more powerful AI and longer to do robot experiments.
Another reason my calc understates the lab’s chance is that there may be higher value from experiments conducted in serial. RoW constructs all their physical capital in parallel in one doubling. Whereas the lab does it over multiple doublings one after the other. That could make a big difference.
How to do better?
The more nuanced way to game this out would be to represent, at each time-step, the physical capital, cognitive labour, and technology (for physical capital) of both the lab and RoW.
Then have a production function for how inputs of physical capital produced and cognitive labour improve technology in each time-step. Probably smg like g_A = (K^a * C^b)^lambda * A^-beta. Where a+b=1, lambda represents the benefits of doing research in serial, and beta accounts for ideas getting harder to find. (This accounts for your other point about the rate of production doubling every 3 doublings. The parameter beta controls whether it’s “3” or some other number.)
Then in each timestep simulate each actor’s change in:
Physical capital
the level of technology determines the physical capital doubling time
so you combine current physical capital and technology to get new physical capital
Technology
Apply the equation above for g_A.
Cognitive labour.
Probably simplest to make this exogenous. Specify some trajectory given your views on the software-only intelligence explosion.
My overall view
I think that physical capital doubling times of a few months shortly after AGI are more plausible than when I first wrote that appendix. So I think this is plausible if we’re starting the clock at the end of the SIE.
Accounting improving serial experiments will favour the lab further. As will the possibility the lab can maintain a cognitive advantage beyond the SIE by designing and quickly manufacturing amazing chips.
But the assumption that a company can spend $1trillion on physical capital before the end of the SIE still seems too high. Though it’s unrealistic to think RoW will use all their physical capital for growth, which balances this out.
The assumption of a 9 month lead feels generous.
Maybe the most plausible scenarios involve ~6 month leads, ~$250b on physical capital, ~1 month robot doubling times at the end of the SIE, and doubling times becoming faster over time.
I made a simple of the dynamic here. (Still not modelling the gains from serial experiments and
So overall I think I’d still bet against the lab being in a position to do this on the economic fundamentals; but it is more plausible that I’d thought.