My co-founders at Anthropic and I were among the first to document and track the “scaling laws” of AI systems—the observation that as we add more compute and training tasks, AI systems get predictably better at essentially every cognitive skill we are able to measure.
The scaling laws paper is only about perplexity, or next-token-prediction loss, not ‘every cognitive skill we are able to measure’. To my knowledge there wasn’t much measurement of compute vs cognitive skills in 2020. IMO we don’t have well-validated scaling laws for compute vs benchmark performance, and any that exist are not a straightforward extension of the 2020 paper.
EDIT: The reason I cared enough about this to comment is that the conflation of these two things launders authority in two ways
The predictability of perplexity vs compute in 2020 applied to present day benchmark performance where labs are much more secretive about training details
The novel contribution of Amodei et al to perplexity scaling, which people care much less about now, makes Dario seem prescient and uniquely qualified to comment on benchmark performance, which everyone cares about in 2026
tl;dr: he was among the first to document and track perplexity scaling. Benchmark scaling is a different thing that (1) he was not among the first to document and track and (2) is not as empirically robust with verifiable open source data.
The scaling laws paper is only about perplexity, or next-token-prediction loss, not ‘every cognitive skill we are able to measure’. To my knowledge there wasn’t much measurement of compute vs cognitive skills in 2020. IMO we don’t have well-validated scaling laws for compute vs benchmark performance, and any that exist are not a straightforward extension of the 2020 paper.
EDIT: The reason I cared enough about this to comment is that the conflation of these two things launders authority in two ways
The predictability of perplexity vs compute in 2020 applied to present day benchmark performance where labs are much more secretive about training details
The novel contribution of Amodei et al to perplexity scaling, which people care much less about now, makes Dario seem prescient and uniquely qualified to comment on benchmark performance, which everyone cares about in 2026
tl;dr: he was among the first to document and track perplexity scaling. Benchmark scaling is a different thing that (1) he was not among the first to document and track and (2) is not as empirically robust with verifiable open source data.