The nature of LLM algorithmic progress

Steven Byrnes5 Feb 2026 19:17 UTC

68 points

There’s a lot of talk about “algorithmic progress” in LLMs, especially in the context of exponentially-improving algorithmic efficiency. For example:

Epoch AI: “[training] compute required to reach a set performance threshold has halved approximately every 8 months”.
Dario Amodei 2025: “I’d guess the number today is maybe ~4x/year”.
Gundlach et al. 2025a “Price of Progress”: “Isolating out open models to control for competition effects and dividing by hardware price declines, we estimate that algorithmic efficiency progress is around 3× per year”.

It’s nice to see three independent sources reach almost exactly the same conclusion—halving times of 8 months, 6 months, and 7½ months respectively. Surely a sign that the conclusion is solid!

…Haha, just kidding! I’ll argue that these three bullet points are hiding three totally different stories. The first two bullets are about training efficiency, and I’ll argue that both are deeply misleading (each for a different reason!). The third is about inference efficiency, which I think is right, and mostly explained by distillation of ever-better frontier models into their “mini” cousins.

Tl;dr / outline

§1 is my attempted big-picture take on what “algorithmic progress” has looked like in LLMs. I split it into four categories:
- §1.1 is stereotypical “algorithmic efficiency improvements” related to the core learning algorithm itself (architectures, tokenizers, optimizers, etc.). I’ll argue that the idea of using a Transformer with optimized hyperparameters was an important idea, but apart from that, the field has produced maybe 3-5× worth of stereotypical “algorithmic efficiency improvements” in the entire period from 2018 to today (≈20%/year). That’s something, but it’s very much less than the exponentials suggested at the top.
- §1.2 is “optimizations”. I figure there might be up to 20× improvement by optimizing the CUDA code, parallelization strategies, precision, and so on, especially if we take the baseline to be a sufficiently early slapdash implementation. But the thing about “optimizations” is that they have a ceiling—it’s not an exponential that can keep growing and growing.
- §1.3 is “data-related improvements”, including proprietary human expert data, and various types of model distillation, both of which have important effects.
- §1.4 is “algorithmic changes that are not really quantifiable as ‘efficiency’”, including RLHF, RLVR, multimodality, and so on. No question that these are important, and we shouldn’t forget that they exist, but they’re not directly related to the exponential-improvement claims at the top.
§2 is why I don’t believe either Epoch AI or Dario, in their claims of exponential training-efficiency improvements (see top).
§3 is a quick sanity-check, studying “nanochat”, which matches GPT-2 performance but costs 600× less to train.
§4 is an optional bonus section on why I care about this topic in the first place. (Not what you expect! Unlike everyone else reading this, I don’t particularly care about forecasting future LLM progress.)

Status of this post

I wrote this very quickly, on a topic where I am not remotely an expert. I’m hoping for feedback and opinions!

1. The big picture of LLM algorithmic progress, as I understand it right now

1.1 Stereotypical “algorithmic efficiency improvements”: there’s the Transformer itself, and … well, actually, not much else to speak of

(This part is heavily reliant on @Hans Gundlach et al. 2025b “On the origin of algorithmic progress in AI”. So I hope it’s right, and that I’m understanding it properly! Please let me know if you have reasons to doubt either of those.)

I’m defining this category as changes related to the core learning algorithm itself—neural architecture changes, SGD vs AdamW, etc. Here’s my current impression:

The replacement of LSTMs with the Transformer in 2017-2018 made a huge difference.
Using the right hyperparameters etc. for the Transformer is also very important, as usual in ML. But duh, everyone knows that! Therefore, the hyperparameters etc. have always been approximately optimal. …Except that there was one mistake which was corrected only in 2022, with the switch from “Kaplan scaling” to “Chinchilla-optimal scaling”.
- Why did the Chinchilla correction take so many years to appear? Probably some combination of: (1) it actually didn’t make that big a difference until the training runs got sufficiently huge, and then it was in fact discovered pretty soon after the discovery became importantly helpful; and (2) the hyperparameter sweep required to surface this error was very expensive, since it involved running a bunch of massive training runs.
Mixture-of-Experts (MoE) was like a 2× efficiency improvement.
All the other legible “algorithmic efficiency improvements” to the Transformer since its invention, at least the ones that Gundlach et al. 2025b looked into—SwiGLU, pre-RMSNorm, rotary encoding, cosine decay learning rate schedule, and improvements to the tokenizer^[1]—adds up to very little, like a factor of less than 1.5× total.^[2]
Presumably there are other “algorithmic efficiency improvements” to the Transformer since its invention that Gundlach et al. 2025b did not look into^[3] (e.g. some might be secret). My current guess (see §2.2 below) is that these all add up in aggregate to, like, very little in the grand scheme of things. Maybe another 2×? Or 3×?

1.2 “Optimizations”: Let’s say up to 20×, but there’s a ceiling

This category is stuff that wouldn’t show up in a FLOP metric, but it’s equally important for cost. It includes everything specific to a particular hardware configuration and training setup—details about quantization, parallelization, FlashAttention, KV cache, other CUDA black magic, and so on, along with system-level optimizations like speculative decoding.

It seems very plausible to me that, after an initial rushed setup of a new configuration, the next many months of configuration-specific optimizations will yield another factor of, I dunno, up to 20×. For example, the original FlashAttention alone apparently sped up some training setups by 3×.

What I don’t believe is that “optimizations” can contribute to an ever-growing exponential, where it’s 10× after two years, 100× after four years, 1000× after six years, etc. These kinds of optimizations have a ceiling, where you’re doing the best you can with the training approach and hardware configuration you’ve got.

1.3 Data-related improvements

As discussed in “Most Algorithmic Progress is Data Progress” by Beren Millidge (@beren), a lot of LLM improvement has come from more and/or better training data,^[4] including:

Paying human experts to create high-quality proprietary training data;
Leveraging AI itself to create high-quality (synthetic) training data, especially by distillation from larger better models to smaller cheaper models, and/or distillation from more “thinking” time to less.
Maybe other things like filtering out bad training data, changing the order that the training data is presented, etc.

What are the impacts of these data-related improvements? Are they relevant to those exponentials at the top? My current take is:

Better data is almost definitely increasing the performance of the best models. (Otherwise companies would not be spending billions of dollars a year on expert human data!) Note that the “performance” in question here is probably benchmarks and applications, but not perplexity.
It follows that better data should also be increasing training efficiency (i.e., decreasing the training compute required to reach any given performance level), at least for the companies that have this kind of proprietary data on hand. But I don’t know quantitatively how big an effect that is.
If we switch topics from training efficiency to inference efficiency, then the point about synthetic data suddenly becomes extremely important: I propose that model distillation is the main explanation for the Gundlach et al. 2025a claim that inference compute has been dropping 3×/year, holding quality fixed. As the biggest and best models get ever bigger and better, the tiny distilled models get better too, thus surpassing quality thresholds that previously required a bigger model.

1.4 Algorithmic changes that are not really quantifiable as “efficiency”

If we put aside the “3×/year” and related quotes at the top, and take a broader view of what LLM algorithmic progress can look like, then of course we find many more items. These include:

RLHF (and DPO, Constitutional AI, etc.);
The rise of long-duration “reasoning” at inference time (and the modifications to training & inference that make this “reasoning” possible—most famously RLVR, but there are claims that certain non-RLVR approaches work equally well (1,2));
Multi-modality;
Tools and interfaces;
Etc.^[5]

2. Explaining away the two training-efficiency exponential claims

At the top, I cited Epoch AI and Dario Amodei as claiming that algorithmic improvements constitute a rapid exponential that’s been going on for years. I don’t currently believe either of them. Here’s why.

2.1 The Epoch “8-month halving time” claim is a weird artifact of their methodology

(The Epoch AI claim in question is at blog, paper, and my response here is entirely based on Gundlach et al. 2025b.)

Some algorithmic changes matter more and more as the model scale gets bigger. Specifically, there were two such changes: the switch from LSTMs to Transformers, and Chinchilla-optimal training.

For example, let’s suppose that the Transformer is N× more efficient than LSTMs with 2018-scale LLMs, and 10N× more efficient than LSTMs with 2025-scale LLMs.

Now let’s put aside everything else, and imagine a world where we switch from LSTMs to Transformers in 2018, and then scale up the transformers from 2018 to 2025 with no additional algorithmic change at all. In the funny Epoch methodology, they would say that we got an “extra” 10× algorithmic improvement (50%/year!) in the 2018-2025 period, because we’re kinda milking ever more advantage from the one-time LSTM-to-Transformer switch.

OK, but … that’s super confusing! Right? By assumption, the algorithms weren’t actually getting better during that 2018-2025 period, at all!

Anyway, the important point is: the actual Epoch analysis seems to be fully compatible with the claims I made in §1.1 above.

Refer to caption — The Gundlach et al. 2025b reanalysis of the supposed exponential efficiency improvement claimed by Epoch AI (2024): it’s mostly just the Transformer itself, and Chinchilla

2.2 The Dario “4x/year” claim is I think just confused

I quoted Dario Amodei 2025 at the top. Here’s a longer version of that quote:

The field is constantly coming up with ideas, large and small, that make things more effective or efficient: it could be an improvement to the architecture of the model (a tweak to the basic Transformer architecture that all of today’s models use) or simply a way of running the model more efficiently on the underlying hardware. New generations of hardware also have the same effect. What this typically does is shift the curve: if the innovation is a 2x “compute multiplier” (CM), then it allows you to get 40% on a coding task for $5M instead of $10M; or 60% for $50M instead of $100M, etc. Every frontier AI company regularly discovers many of these CM’s: frequently small ones (~1.2x), sometimes medium-sized ones (~2x), and every once in a while very large ones (~10x). … In 2020, my team published a paper suggesting that the shift in the curve due to algorithmic progress is ~1.68x/year. That has probably sped up significantly since; it also doesn’t take efficiency and hardware into account. I’d guess the number today is maybe ~4x/year. Another estimate is here.

At first, I found this quote baffling:

If we’re talking about “algorithmic” changes as in §1.1 … like, man, I remember how excited people were about rotary encoding, and MoE, when those were first used. Am I supposed to believe that the last five years have had dozens of rotary-encoding-level algorithmic advances, and maybe 10 MoE-level algorithmic advances, and a handful of way-bigger-than-MoE-level algorithmic advances, all of which have stayed secret for years? No way.
If we’re talking about “optimization” changes as in §1.2, I mean, Dario’s been working primarily on Transformer-based LLMs for at least 7 years. I flat out do not believe that those kinds of “optimizations” can make the same calculations run more economically by a factor of $4^{7}$ = 16,000× (or whatever), nor anywhere remotely close to that. There’s a ceiling. GPU utilization cannot exceed 100%. Quantization can’t go below 1 bit. Etc.

So what the heck is Dario talking about??

…But I think I got it now. So here’s my current take—the only way I can get everything to fit together and make sense:

Maybe some of Dario’s “compute multipliers” are actually part of the category of “data-related improvements” (§1.3 above) (i.e., expert human data, different types of model distillation, etc.). I mean, it doesn’t really sound that way, from what he wrote, but sure, maybe.
A few of Dario’s “compute multipliers” are the ones listed in §1.1, such as MoE. But that’s not very much, if we’re ultimately trying to explain a total multiplier in the ballpark of 4×/year since 2018 = 16,000×.
Approximately all the rest of Dario’s “compute multipliers” are in the category of “optimizations” (§1.2 above).
…And my best guess right now is: I think Dario is excited by the fact that his staff keep finding new optimizations and telling him about it, and he is getting the idea that these optimizations keep piling up without any ceiling. But he’s confused! The optimizations definitely do have a ceiling! It’s just that Anthropic is periodically switching to new chips / new chip configurations / bigger model size / new training setups / etc., and then picking the low-hanging optimization fruit from this new tree. Thus, Anthropic staff are indeed producing a never-ending fountain of optimizations, and telling Dario about them, but these are not the kinds of improvements that pile up into a steadily-accumulating exponential. Rather, you need to stop using the old optimizations when you switch to a new configuration.

So overall, I think Dario is somewhat confused, and giving a misleading (or at least very confusing) description of the situation.

[Boy, I sure feel weird about lecturing Dario Amodei on the big picture of LLM training! He knows more about LLM training than almost anyone on Earth, and I have (checks notes) no LLM training experience whatsoever. So if anyone has a different proposal for how to make sense of Dario’s quote above, I’m all ears!]

3. Sanity-check: nanochat

There were seven years between GPT-2 (trained in early 2019) and the new nanochat, which matches GPT-2 performance on the “CORE” metric (“a diverse set of reasoning and knowledge tasks from the DCLM benchmark suite”).

Remarkably, Karpathy says here that nanochat training costs $73 (“3 hours on a single 8XH100 node”), whereas “GPT-2 was trained by OpenAI on 32 TPU v3 chips for 168 hours (7 days), with $8/hour/TPUv3 back then, for a total cost of approx. $43K”.

So that would be a factor of 600 in 7 years, i.e. a halving time of 9 months. Is that consistent with my story in §1? I think so! For example, it might be something like:

6× from lower hardware costs (Epoch says FLOP/$ has increased 30%/year, and ${1.3}^{7}$ =6);
2× from “algorithmic efficiency improvements” (§1.1);
10× from “optimizations” (§1.2);
- (This category would include not only FlashAttention etc., but also GPU hardware changes since 2019 that allow for higher utilization.)
5× from better data (§1.3)

For that last one: GPT-2 used “webtext”, which was generated by scraping URLs linked from Reddit. By contrast, nanochat trains on “fineweb-EDU”, a dataset of educational materials crafted and curated with incomparably more effort and care. Remember, we’re comparing nanochat to GPT-2 on “reasoning and knowledge tasks”, not perplexity; I would be shocked if this better data was not playing a major role.

So anyway, my take in §1 seems at least plausibly consistent with the nanochat thing, AFAICT at a glance. To be clear, I didn’t check things in detail or scrutinize it very hard. If anyone wants to really check, you could just download nanochat and have at it!

4. Optional bonus section: why does this matter?

Well for 99% of the people reading this post, this topic matters to you because you’re trying to forecast future LLM progress. But that’s not my interest, so I won’t talk about it. I’ll leave that to others!

I’m actually interested in a rather different debate, related to arguments about takeoff speeds for a hypothetical future AI paradigm—see Foom & Doom 1: “Brain in a box in a basement”, e.g. comment on that post by @ryan_greenblatt. Here’s the debate:

One school of thought (that I vaguely associate with Paul Christiano^[6]) says: When people are trying to do something in ML, they very rapidly get to near the ceiling of how efficiently they can do that thing, given the available data and hardware situation (but perhaps leaving aside paradigm shifts, which are not about doing the same thing more efficiently, but rather about trying to do something different instead).

A different school of thought says: No, that’s wrong, instead when people are trying to do something in ML, there will be a very large collection of algorithmic improvements that could make it work more efficiently, and these improvements will take many thousands of person-years to discover, and they will collectively amount to orders of magnitude of efficiency difference.

I’m generally in the first school of thought, which of course goes along with my belief that a future AI paradigm shift could lead to a remarkably sudden emergence of AGI and ASI.

…OK, if that’s the debate, then what lesson do we take away from this LLM case-study? My answer: If I’m right (a big “if”!), then the history of LLMs seems to mostly support the first school of thought.

To be clear, I don’t think this kind of analogizing is terribly strong evidence either way; and note also that there are other case-studies like this ImageNet analysis that might or might not paint a different picture, I didn’t check.

In fact, there are two key disanalogies between LLMs versus the future AI paradigm I’m expecting (see Foom & Doom 1), and they make my case for next-paradigm sudden takeoff even stronger: the future paradigm I’m expecting would (1) not rely on training data for its capabilities (unlike LLMs), making §1.3 basically moot; and (2) require very little compute to get from random initialization to AGI (if efficiently implemented), which would allow for much more rapid iteration and testing than we’re used to from LLMs.

Thanks Hans Gundlach, Seth Herd, plex, algon, and ishaan for critical comments on earlier drafts.

^
Gundlach links to this paper on tokenizers, and describes it as a claim that “inefficient tokenization” can cause up to 68% performance hit. But I think that 68% number comes from comparing best practices to the extraordinarily dumb idea of building a tokenizer using English-only text and then running inference on a multilingual corpus. As for real tokenizer improvements, I think everyone’s been using BPE since before the Transformer, and different flavors of BPE seem quite similar, if I’m reading the paper right. As an example, this page benchmarks the tokenizer of the recently-released nanochat (§3) against the tokenizer used by GPT-2 in 2019, and finds 0-15% compression difference, depending on the data type. The difference probably comes from using better training data to set up the tokenizer.
^
FYI: They note that if you revert one of these things at a time, it has a bigger deleterious impact than if you revert the whole package at once. In other words, the “Retro Transformer” and the “Modern Transformer” were each a package of components that worked particularly well together.
^
For example, Karpathy here mentions the “muon optimizer … residual pathways and skip connections gated by learnable scalars, and value embeddings”, none of which seem to be studied by Gundlach et al. 2025b, unless I missed it.
^
This of course fits perfectly with my belief that we should think of LLMs as getting their impressive powers almost entirely via imitation learning from their training corpus.
^
Edited to add: Oh here’s another thing: context window extension (e.g YaRN). I guess my take is: this could go either in this category (it’s arguably an add-on that allows an LLM to do something new), or in §1.1 (it’s arguably a more-efficient replacement for training long context windows in the naïve way). I dunno.
^
For example, Paul Christiano 2018: “A more precise version [of] my claim: if you gave smart grad students from 1990 access to all of the non-AI technology of 2017 (esp. software tools + hardware + data) and a big budget, it would not take them long to reach nearly state of the art performance on supervised learning and RL. For example, I think it’s pretty plausible that 20 good grad students could do it in 3 years if they were motivated and reasonably well managed.” (Actually, Paul is much further in the direction of the first school of thought than I am, because I defined it with a carve-out for possible hard-to-discover paradigm shifts, and he’s not even conceding that.)