My sense is that the AI 2040 authors underrate the criticisms I’ve raised above in large part because they expect superintelligence so imminently. My third criticism of AI 2040 is that it buys too uncritically into the idea of a sharp takeoff of AI capabilities. I’m not denying the possibility that this could occur in principle. And it’s true that the last decade (and especially the last 5 years) of AI capabilities progress have been blindingly fast in most measurable ways—far faster than almost anyone (except a few prescient forecasters like Legg, Amodei, Kokotajlo, Leike, and Kurzweil) predicted. However, the real-world impacts of AI (aside from the ballooning revenues of AI companies) have been underwhelming thus far, especially when compared to the progress in measured capabilities.
I’m actually sympathetic to your criticisms, as you know from personal conversations. But I am very unsympathetic to this thing you say here about us buying uncritically into sharp takeoff of AI capabilities.
First of all, I’ve made a lot of predictions about AI capabilities and their diffusion and effects on society over the past years—going back at least to What 2026 Looks Like in 2021 -- more than almost anyone else. And I’ve been overall pretty accurate. It’s true that W2026LL overestimated the impacts of some things—most notably AI-powered persuasion—but it’s also true that I underestimated the impacts of other things, e.g. AI company revenue has grown faster than AI 2027 predicted and GDP growth contribution from AI has been more than W2026LL and AI 2027 predicted.
Also, isn’t deploying AIs internally to accelerate AI R&D an example of a real-world impact? It’s probably the most important kind of impact actually. And it sure is happening faster than almost everyone else expected and potentially faster than even I expected.
the “divergence between capabilities and impacts’ we observe, with specific reference to AI2027 predictions for example, seems to be that AI has turned out to be better aligned than we expected. Even with the rise of ~superhuman coder level AI, we’ve seen virtually no ill effects so far (a small uptick in crypto and ransomware attacks aside).
I think progress in alignment research has not been adequately used to update your priors on RSI being most likely misaligned.
I think your forecasting has been absolutely stellar, but surely you can do a forecast on how likely this wishlist scenario is, and would arrive at a number too low to even seem worth thinking about it, no? Even imagining the ultrarational US administration required should be too low to think about, let alone when you add all of the rest.
It seems like even a super basic ‘We use all the might we can gather to convince one by one all the biggest actors seperately to delay and put more into Safety, and do lesser more plausible regulations that incentivize using more of their compute towards Safety in a way all labs benefit from’ or some slightly improved version looks significantly more plausible, no? (still not very plausible, maybe not as pie in the sky at least but still pie in the troposphere).
Or better yet, is there by any chance a realistic plan that you are about to follow-up AI 2040 with?
Edit: I guess you’ve already outlined your Plan B and it’s already ‘Fight China’..
I’m actually sympathetic to your criticisms, as you know from personal conversations. But I am very unsympathetic to this thing you say here about us buying uncritically into sharp takeoff of AI capabilities.
First of all, I’ve made a lot of predictions about AI capabilities and their diffusion and effects on society over the past years—going back at least to What 2026 Looks Like in 2021 -- more than almost anyone else. And I’ve been overall pretty accurate. It’s true that W2026LL overestimated the impacts of some things—most notably AI-powered persuasion—but it’s also true that I underestimated the impacts of other things, e.g. AI company revenue has grown faster than AI 2027 predicted and GDP growth contribution from AI has been more than W2026LL and AI 2027 predicted.
Also, isn’t deploying AIs internally to accelerate AI R&D an example of a real-world impact? It’s probably the most important kind of impact actually. And it sure is happening faster than almost everyone else expected and potentially faster than even I expected.
the “divergence between capabilities and impacts’ we observe, with specific reference to AI2027 predictions for example, seems to be that AI has turned out to be better aligned than we expected. Even with the rise of ~superhuman coder level AI, we’ve seen virtually no ill effects so far (a small uptick in crypto and ransomware attacks aside).
I think progress in alignment research has not been adequately used to update your priors on RSI being most likely misaligned.
(also there is the AI futures model and all the preceding attempts to model takeoff)
I think your forecasting has been absolutely stellar, but surely you can do a forecast on how likely this wishlist scenario is, and would arrive at a number too low to even seem worth thinking about it, no? Even imagining the ultrarational US administration required should be too low to think about, let alone when you add all of the rest.
It seems like even a super basic ‘We use all the might we can gather to convince one by one all the biggest actors seperately to delay and put more into Safety, and do lesser more plausible regulations that incentivize using more of their compute towards Safety in a way all labs benefit from’ or some slightly improved version looks significantly more plausible, no? (still not very plausible, maybe not as pie in the sky at least but still pie in the troposphere).
Or better yet, is there by any chance a realistic plan that you are about to follow-up AI 2040 with?
Edit: I guess you’ve already outlined your Plan B and it’s already ‘Fight China’..