Yeah, i think one of the biggest weaknesses of this model, and honestly of most thinking on the intelligence explosion, is not carefully thinking through the data.
During an SIE, AIs will need to generate data themselves, by doing the things that human researchers currently do to generate data. That includes finding new untapped data sources, creating virtual envs, creating SFT data themselves by doing tasks with scaffolds, etc.
OTOH it seems unlikely they’ll have anything as easy as the internet to work with. OTOH, internet data is actually v poorly targeted at teaching AIs how to do crucial real-world tasks, so perhaps with abundant cognitive labour you can do much better and make curriculla that directly targeted the skills that most need improving
Thanks! (Quickly written reply!)
I believe I was here thinking about how society has, at least in the past few hundred years, spent a minority of GDP on obtaining new raw materials. Which suggests that access to such materials wasn’t a significant bottleneck on expansion.
So it’s a stronger claim that “hard cap”. I think a hard cap would, theoretically, result in all GDP being used to unblock the bottleneck, as there’s no other way to increase GDP. I think you could quantify the strength of the bottleneck as the marginal elasticity of GDP to additional raw materials. In a task-based model, i think the % of GDP spent on each task is proportional to this elasticity?
Yeah, I think maybe it is? I do feel like, given the very long history of sustained growth, it’s on the sceptic to explain why their proposed bottleneck will kick in with explosive growth but not before. So you could state my argument as: raw materials never bottlenecked growth before; no particular reason they would just bc growth is faster bc that faster growth is driven by having more labour+capital which can be used for gathering more resources; so we shouldn’t expect raw materials to bottleneck growth in the future.
TBC, this is all compatible with “if we had way more raw materials then this would boost output”. E.g. in Cobb Douglas doubling an input to output increases output notably, but there still aren’t bottlenecks.
(And i actually agree that it’s more like CES with rho<0, i.e. raw materials is a stronger bottleneck, but just think we’ll be able to spend output to get more raw materials.)
(Also, to clarify: this is all about the feasibility of explosive growth. I’m not claiming it would be good to do any of this!)