A GPT4 scale model trained on the dataset of GPT3 would be substantially worse across all benchmarks, even if we somehow replicated the GPT3-dataset to be the scale of GPT4s dataset.
My pet theory is that’s what basically happened with Llama 4 Behemoth. Unfortunately, I don’t know how to verify or falsify it.
However, there is a counterargument to be made to Beren’s thesis: namely, the NanoGPT speedrun. Granted, it only started in May 2024, but as noted by Tamay Besiroglu last February (he in turn expands on ideas and methods from a 2023 Epoch AI paper on game speedruns), the speed increases as a power law with the number of tries (not the calendar time!).
The trend line is still mostly the same as in his tweet, although I ran a QLR/sup-Wald test for structural breaks to be sure, and detected two slightly different periods:
one before the introduction of FlexAttention in November 2024, which caused a statistically significant jump down (visible on the chart in the tweet);
and another for all of the records since (actually most of the data, 46 points out of 58 as of this writing).
I also redid the analysis for the successive record ratios with the new data, the trend is basically the same despite the noise and scatter but QLR appears to not work properly because of timing rule changes last January.
I would have posted the charts here but don’t know how to do that (just I couldn’t figure out how to ping Beren!), and one can redo the analysis with an LLM or a coding agent anyway.
These results imply that non-data algorithmic progress for pretraining specifically definitely exists (the speedup is ~25x in under 2 years) but may be a bit slowing down over time.
Thanks, a very interesting post, sounds quite convincing!
Checking the source, he wrote:
My pet theory is that’s what basically happened with Llama 4 Behemoth. Unfortunately, I don’t know how to verify or falsify it.
However, there is a counterargument to be made to Beren’s thesis: namely, the NanoGPT speedrun. Granted, it only started in May 2024, but as noted by Tamay Besiroglu last February (he in turn expands on ideas and methods from a 2023 Epoch AI paper on game speedruns), the speed increases as a power law with the number of tries (not the calendar time!).
The trend line is still mostly the same as in his tweet, although I ran a QLR/sup-Wald test for structural breaks to be sure, and detected two slightly different periods:
one before the introduction of FlexAttention in November 2024, which caused a statistically significant jump down (visible on the chart in the tweet);
and another for all of the records since (actually most of the data, 46 points out of 58 as of this writing).
I also redid the analysis for the successive record ratios with the new data, the trend is basically the same despite the noise and scatter but QLR appears to not work properly because of timing rule changes last January.
I would have posted the charts here but don’t know how to do that (just I couldn’t figure out how to ping Beren!), and one can redo the analysis with an LLM or a coding agent anyway.
These results imply that non-data algorithmic progress for pretraining specifically definitely exists (the speedup is ~25x in under 2 years) but may be a bit slowing down over time.