That is, scale-independent algorithmic improvements give a constant compute-equivalent-multiplier at 10^15 FLOPs and at 10^24 FLOPs; a constant ~2x multiplier would raise effective training compute from 1x10^N to 2x10^N no matter what N is. Prior works often acted as if this is what an “algorithmic improvement” meant—the kind of thing that acted as a constant multiplier of compute. But scale-dependent algorithmic improvements give variable compute-equivalent multipliers when moving from 10^15 FLOPs to 10^25 FLOPs—they could give OOM larger gains at larger sizes.
Thus, Gundlach argues that a great deal of apparent “algorithmic progress” is an artifact of analyzing unchanging algorithms -- whose performance increases as FLOPs increase—in a world where the FLOPs for training LLMs is continuously increasing. So almost all of the apparent progress is due to moving from LSTMs to Transformers (2017) and from Kaplan to Chinchilla optimization (2022). The measured algorithmic improvement then mostly doesn’t reflect continued a flow of improved “algorithms” but a small stock of algorithms that keep improving relative to their predecessors as FLOPs in training runs increases.
Put otherwise, if counterfactually the size of the largest training runs had not been continuously increasing over time, the {”flow of algorithms” model / all algorithms are scale-independent model} predicts efficiency improvements would have kept going at about the same rate; while the {”scale-dependent algorithm stock” model / most important algorithms are scale-dependent} predicts efficiency improvements would have slowed almost to a halt. Both Anson Ho and Steven Byrnes seem to find this account at least somewhat right.
But although this paper finds scale-dependent algorithms to be very influential, I think it might be actually understating its case about the importance of scale-dependent algorithms!
Gundlach looks at several mixture-of-experts papers, finds that they report compute-scaling of around 2x, and then reports a scale independent 2x gain from MoE (they note that this is provisional). But several unexamined papers explicitly report large-ish scale dependent compute multipliers.
“Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models” finds a formula for “efficiency leverage” and claims that in “large-scale pre-training scenario[s]” MoE “efficiency gains become increasingly significant as computational resources expand.” They predict a > 7x efficiency gain at 1e22 scale, and more as you scale up.
In general, I find claims in this apprximate ballpark of scale-dependent algorithmic improvement (of around ~8x at the current frontier) to be more likely than a fixed 2x or 3x improvement. Why?
These papers are explicitly about scaling laws
These papers use the most modern highly-granual, small-expert style of MoE, which are what you want to actually look at when trying to determine what kind of scaling laws are going on.
(Strong, vague heuristic): The human brain is really sparse, my guess is that sparsity in general is such a ubiquitous feature of intelligence-like stuff that it would be weird if our math didn’t find it to be fundamental with favorable scaling laws.
If this is true, then it seems possible that Gundlach’s ~90% of gains coming from scale-dependent algorithms might be even too conservative? Which would be surprising to me.
And to the degree scale-dependent gains indicate that a software-only intelligence explosion is unlikely, this would further decrease the likelihood of a SIE; although by a very modest amount and leaving a great deal uncertain.
The clearest source I know on compute multipliers of MoE is this Jan 2025 paper. The key data is in Appendix D.4, Figure 11 and Figure 12, left. My takeaway is that 8x sparsity gives a 3x compute multiplier, and 30x sparsity gives a 6x compute multiplier. Figure 11 suggests that the gains continue with more sparsity, and are linear in the logarithm of sparsity (but then we run out of data, since MoEs also increase the compute optimal D/N ratio, seemingly by exactly the same factor as the compute multiplier boon).
(I make a couple more points on why it’s hard to get a sense for these numbers from indirect estimates, experiments that are not specifically about this particular question, in a comment I left under the “Origin of Algorithmic Progress” linkpost 6 months ago.)
Note: MoEs Probably Give Large, Scale-Dependent Improvements
“On the Origin of Algorithmic Progress in AI” (2025) plausibly claims that a great deal of the apparent algorithmic progress in LLMs -- 22,000x from 2015 to 2023 -- has been sort of fake, due to increased compute using scale-dependent algorithms.
That is, scale-independent algorithmic improvements give a constant compute-equivalent-multiplier at 10^15 FLOPs and at 10^24 FLOPs; a constant ~2x multiplier would raise effective training compute from 1x10^N to 2x10^N no matter what N is. Prior works often acted as if this is what an “algorithmic improvement” meant—the kind of thing that acted as a constant multiplier of compute. But scale-dependent algorithmic improvements give variable compute-equivalent multipliers when moving from 10^15 FLOPs to 10^25 FLOPs—they could give OOM larger gains at larger sizes.
Thus, Gundlach argues that a great deal of apparent “algorithmic progress” is an artifact of analyzing unchanging algorithms -- whose performance increases as FLOPs increase—in a world where the FLOPs for training LLMs is continuously increasing. So almost all of the apparent progress is due to moving from LSTMs to Transformers (2017) and from Kaplan to Chinchilla optimization (2022). The measured algorithmic improvement then mostly doesn’t reflect continued a flow of improved “algorithms” but a small stock of algorithms that keep improving relative to their predecessors as FLOPs in training runs increases.
Put otherwise, if counterfactually the size of the largest training runs had not been continuously increasing over time, the {”flow of algorithms” model / all algorithms are scale-independent model} predicts efficiency improvements would have kept going at about the same rate; while the {”scale-dependent algorithm stock” model / most important algorithms are scale-dependent} predicts efficiency improvements would have slowed almost to a halt. Both Anson Ho and Steven Byrnes seem to find this account at least somewhat right.
But although this paper finds scale-dependent algorithms to be very influential, I think it might be actually understating its case about the importance of scale-dependent algorithms!
Gundlach looks at several mixture-of-experts papers, finds that they report compute-scaling of around 2x, and then reports a scale independent 2x gain from MoE (they note that this is provisional). But several unexamined papers explicitly report large-ish scale dependent compute multipliers.
“Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models” finds a formula for “efficiency leverage” and claims that in “large-scale pre-training scenario[s]” MoE “efficiency gains become increasingly significant as computational resources expand.” They predict a > 7x efficiency gain at 1e22 scale, and more as you scale up.
“Scaling Laws for Fine-Grained Mixture of Experts” predicts steeper gains, claiming a 20x increase at a mere 10^20 FLOPs. (This is an older paper and my tentative guess is this is too large.)
In general, I find claims in this apprximate ballpark of scale-dependent algorithmic improvement (of around ~8x at the current frontier) to be more likely than a fixed 2x or 3x improvement. Why?
These papers are explicitly about scaling laws
These papers use the most modern highly-granual, small-expert style of MoE, which are what you want to actually look at when trying to determine what kind of scaling laws are going on.
(Strong, vague heuristic): The human brain is really sparse, my guess is that sparsity in general is such a ubiquitous feature of intelligence-like stuff that it would be weird if our math didn’t find it to be fundamental with favorable scaling laws.
If this is true, then it seems possible that Gundlach’s ~90% of gains coming from scale-dependent algorithms might be even too conservative? Which would be surprising to me.
And to the degree scale-dependent gains indicate that a software-only intelligence explosion is unlikely, this would further decrease the likelihood of a SIE; although by a very modest amount and leaving a great deal uncertain.
The clearest source I know on compute multipliers of MoE is this Jan 2025 paper. The key data is in Appendix D.4, Figure 11 and Figure 12, left. My takeaway is that 8x sparsity gives a 3x compute multiplier, and 30x sparsity gives a 6x compute multiplier. Figure 11 suggests that the gains continue with more sparsity, and are linear in the logarithm of sparsity (but then we run out of data, since MoEs also increase the compute optimal D/N ratio, seemingly by exactly the same factor as the compute multiplier boon).
(I make a couple more points on why it’s hard to get a sense for these numbers from indirect estimates, experiments that are not specifically about this particular question, in a comment I left under the “Origin of Algorithmic Progress” linkpost 6 months ago.)