Imagine going 10 or 20 years into the past, telling people a selection of benchmark scores of current AIs, and asking them to predict what the world that contains them looks like. I expect that they would have described a world that was dramatically transformed—perhaps one in which AIs already wielded enormous political power, or had made far-reaching scientific breakthroughs, or at the very least had decimated white-collar jobs.
[...]
However, regardless of the specific cause of the great divergence between capabilities and impacts, it’s something which deserves to be grappled with more directly.
How confident are you in that statement?
If you told them something like “AIs have an 80%-reliability time horizon of 3 hours on software engineering evaluation tasks in mid 2026 (and cross 1 hour in early 2026)” I think they would have predicted something roughly like the current $100B of coding revenue from AI companies. And I don’t think they would have predicted any of the outcomes you listed.
I agree some other benchmarks that are more expertise-heavy (e.g. IMO math, MMLU, etc.) ended up saturated way faster than people expected, but I think that’s just downstream of a mistake people made where they underestimated how easy it would be to build somewhat narrow systems that solve these sorts of small-horizon natural language problems, and not the symptom of a disconnect between measured capabilities and impact.
Maybe a disconnect will exist in the future because measuring very long horizon capabilities on the tasks we care most about is difficult, but this feels more like “people will suck at building good capabilities benchmarks” than “there will be a mysterious gap that deserves to be grappled with”.
I think that’s just downstream of a mistake people made where they underestimated how easy it would be to build somewhat narrow systems that solve these sorts of small-horizon natural language problems
But the systems that are saturating IMO math, MMLU, etc, are not very narrow in the sense that people would have used the term a decade or two ago. So you can think about their mistake as the inability to imagine systems which have become much less narrow (to the extent that a single system can saturate almost all language and vision benchmarks from a decade ago) but are still far away from taking over the world. Then the question is: why not expect the same to happen over the next decade?
How confident are you in that statement?
If you told them something like “AIs have an 80%-reliability time horizon of 3 hours on software engineering evaluation tasks in mid 2026 (and cross 1 hour in early 2026)” I think they would have predicted something roughly like the current $100B of coding revenue from AI companies. And I don’t think they would have predicted any of the outcomes you listed.
I agree some other benchmarks that are more expertise-heavy (e.g. IMO math, MMLU, etc.) ended up saturated way faster than people expected, but I think that’s just downstream of a mistake people made where they underestimated how easy it would be to build somewhat narrow systems that solve these sorts of small-horizon natural language problems, and not the symptom of a disconnect between measured capabilities and impact.
Maybe a disconnect will exist in the future because measuring very long horizon capabilities on the tasks we care most about is difficult, but this feels more like “people will suck at building good capabilities benchmarks” than “there will be a mysterious gap that deserves to be grappled with”.
But the systems that are saturating IMO math, MMLU, etc, are not very narrow in the sense that people would have used the term a decade or two ago. So you can think about their mistake as the inability to imagine systems which have become much less narrow (to the extent that a single system can saturate almost all language and vision benchmarks from a decade ago) but are still far away from taking over the world. Then the question is: why not expect the same to happen over the next decade?