For future work on the software intelligence explosion, I’d like to see 2 particular points focused on here, @Tom Davidson:
1 is estimating the complementarity issue, and more generally pinning down the rho number for software, because whether complementary or substitution effects dominate during the lead up to automating all AI R&D is a huge factor in whether an intelligence explosion is self-sustaining.
More from Tamay Besiroglu and Natalia Coelho here:
Tamay Besiroglu: One interesting fact is that there’s a strong coincidence in the rate of software progress and hardware scaling both pre- and post the deep learning era in also in other domains of software. That observation seems like evidence of complementarities.
Natalia Coelho: (Just to clarify for those reading this thread) “0.45-0.87” is this paper’s estimate for sigma (the elasticity of substitution) not rho (the substitution parameter). So this indicates complementarity between labor and capital. The equivalent range for rho is −1.22 to −0.15
Thus, we can use the −1.22 to −0.15 as a base rate for rho for software, and argue for that number being up or down based on good evidence/arguments for the number being higher or lower.
My second ask is to get more information on the value of r, which is the returns on software, because right now the numbers you have gotten are way too uncertain for it to be much use (especially for predicting how fast an AI can improve), and I’d like to see more work on estimating the parameter r using many sources of data.
A question: How hard is it to actually figure out if compute scaling is driving most of the progress, compared to algorithms.
You mentioned it’s very hard to decompose the variables, but is it the sort of thing you’d need like a 6 month project for, or would we just have to wait several years for things to play out, because if it could be predicted before it happens, it would be very, very valuable evidence.
Tom Davidson: Yeah would be great to tease this apart. But hard:
- Hard to disentangle compute for experiments from training+inference.
- V hard to attribute progress to compute vs researchers when both rise together
For future work on the software intelligence explosion, I’d like to see 2 particular points focused on here, @Tom Davidson:
1 is estimating the complementarity issue, and more generally pinning down the rho number for software, because whether complementary or substitution effects dominate during the lead up to automating all AI R&D is a huge factor in whether an intelligence explosion is self-sustaining.
More from Tamay Besiroglu and Natalia Coelho here:
https://x.com/tamaybes/status/1905435995107197082
https://x.com/natalia__coelho/status/1906150456302432647
Thus, we can use the −1.22 to −0.15 as a base rate for rho for software, and argue for that number being up or down based on good evidence/arguments for the number being higher or lower.
My second ask is to get more information on the value of r, which is the returns on software, because right now the numbers you have gotten are way too uncertain for it to be much use (especially for predicting how fast an AI can improve), and I’d like to see more work on estimating the parameter r using many sources of data.
A question: How hard is it to actually figure out if compute scaling is driving most of the progress, compared to algorithms.
You mentioned it’s very hard to decompose the variables, but is it the sort of thing you’d need like a 6 month project for, or would we just have to wait several years for things to play out, because if it could be predicted before it happens, it would be very, very valuable evidence.
Tweet below:
https://x.com/TomDavidsonX/status/1905905058065109206