Yes, though I’m much more comfortable explaining and arguing for my own position than EY’s. It’s just that my position turns out to be pretty similar. (Partly this is independent convergence, but of course partly this is causal influence since I’ve read a lot of his stuff.)
There’s a lot to talk about, I’m not sure where to begin, and also a proper response would be a whole research project in itself. Fortunately I’ve already written a bunch of it; see these twosequences.
Here are some quick high-level thoughts:
1. Begin with timelines. The best way to forecast timelines IMO is Ajeya’s model; it should be the starting point and everything else should be adjustments from it. The core part of Ajeya’s model is a probability distribution over how many OOMs of compute we’d need with today’s ideas to get to TAI / AGI / APS-AI / AI-PONR / etc. [Unfamiliar with these acronyms? See Robbo’s helpful comment below] For reasons which I’ve explained in my sequence (and summarized in a gdoc) my distribution has significantly more mass on the 0-6 OOM range than Paul does, and less on the 13+ range. The single post that conveys this intuition most is Fun with +12 OOMs.
Now consider how takeoff speed views interact with timelines views. Paul-slow takeoff and <10 year timelines are in tension with each other. If <7 OOMs of compute would be enough to get something crazy powerful with today’s ideas, then the AI industry is not an efficient market right now. If we get human-level AGI in 2030, then on Paul’s view that means the world economy should be doubling in 2029 and should have doubled over the course of 2025 − 2028 and should already be accelerating now probably. It doesn’t look like that’s happening or about to happen. I think Paul agrees with this; in various conversations he’s said things like “If AGI happens in 10 years or less then probably we get fast takeoff.” [Paul please correct me if I’m mischaracterizing your view!]
Ajeya (and Paul) mostly update against <10 year timelines for this reason. I, by contrast, mostly update against slow takeoff. (Obviously with both do a bit of both, like good Bayesians.)
2. I feel like the debate between EY and Paul (and the broader debate about fast vs. slow takeoff) has been frustratingly much reference class tennis and frustratingly little gears-level modelling. This includes my own writing on the subject—lots of historical analogies and whatnot. I’ve tentatively attempted some things sorta like gears-level modelling (arguably What 2026 Looks Like is an example of this) and so far it seems to be pushing my intuitions more towards “Yep, fast takeoff is more likely.” But I feel like my thinking on this is super inadequate and I think we all should be doing better. Shame! Shame on all of us!
3. I think the focus on GDP (especially GWP) is really off, for reasons mentioned here. I think AI-PONR will probably come before GWP accelerates, and at any rate what we care about for timelines and takeoff speeds is AI-PONR and so our arguments should be about e.g. whether there will be warning shots and powerful AI tools of the sort that are relevant to solving alignment for APS-AI systems.
I feel like the debate between EY and Paul (and the broader debate about fast vs. slow takeoff) has been frustratingly much reference class tennis and frustratingly little gears-level modelling.
So, there’s this inherent problem with deep gearsy models, where you have to convey a bunch of upstream gears (and the evidence supporting them) before talking about the downstream questions of interest, because if you work backwards then peoples’ brains run out of stack space and they lose track of the whole multi-step path. But if you just go explaining upstream gears first, then people won’t immediately see how they’re relevant to alignment or timelines or whatever, and then lots of people just wander off. Then you go try to explain something about alignment or timelines or whatever, using an argument which relies on those upstream gears, and it goes right over a bunch of peoples’ heads because they don’t have that upstream gear in their world-models.
For the sort of argument in this post, it’s even worse, because a lot of people aren’t even explicitly aware that the relevant type of gear is a thing, or how to think about it beyond a rough intuitive level.
I first ran into this problem in the context of takeoff arguments a couple years ago, and wrote up this sequence mainly to convey the relevant kinds of gears and how to think about them. I claim that this (i.e. constraint slackness/tautness) is usually a good model for gear-type in arguments about reference-classes in practice: typically an intuitively-natural reference class is a set of cases which share some common constraint, and the examples in the reference class then provide evidence for the tautness/slackness of the constraint. For instance, in this post, Paul often points to market efficiency as a taut constraint, and Eliezer argues that constraint is not very taut (at least not in the way needed for the slow takeoff argument). Paul’s intuitive estimates of tautness are presumably driven by things like e.g. financial markets. On the other side, Eliezer wrote Inadequate Equilibria to talk about how taut market efficiency is in general, including gears “further up” and more examples.
So, there’s this inherent problem with deep gearsy models, where you have to convey a bunch of upstream gears (and the evidence supporting them) before talking about the downstream questions of interest, because if you work backwards then peoples’ brains run out of stack space and they lose track of the whole multi-step path. But if you just go explaining upstream gears first, then people won’t immediately see how they’re relevant to alignment or timelines or whatever, and then lots of people just wander off. Then you go try to explain something about alignment or timelines or whatever, using an argument which relies on those upstream gears, and it goes right over a bunch of peoples’ heads because they don’t have that upstream gear in their world-models.
The solution might be to start with a concise, low-detail summery (not even one that argues the case, just states it), then start explaining in full detail from the start, knowing that your readers now know which way you’re going.
Wait, I think I just invented the Abstract (not meant as a snide remark. I really did realize it after writing the above, and found it funny).
The core part of Ajeya’s model is a probability distribution over how many OOMs of compute we’d need with today’s ideas to get to TAI / AGI / APS-AI / AI-PONR / etc.
I didn’t know the last two acronyms despite reading a decent amount of this literature, so thought I’d leave this note for other readers. Listing all of them for completeness (readers will of course know the first two):
TAI: transformative AI
AGI: artificial general intelligence
APS-AI: Advanced, Planning, Strategically aware AI [1]
Yes, though I’m much more comfortable explaining and arguing for my own position than EY’s. It’s just that my position turns out to be pretty similar. (Partly this is independent convergence, but of course partly this is causal influence since I’ve read a lot of his stuff.)
There’s a lot to talk about, I’m not sure where to begin, and also a proper response would be a whole research project in itself. Fortunately I’ve already written a bunch of it; see these two sequences.
Here are some quick high-level thoughts:
1. Begin with timelines. The best way to forecast timelines IMO is Ajeya’s model; it should be the starting point and everything else should be adjustments from it. The core part of Ajeya’s model is a probability distribution over how many OOMs of compute we’d need with today’s ideas to get to TAI / AGI / APS-AI / AI-PONR / etc. [Unfamiliar with these acronyms? See Robbo’s helpful comment below] For reasons which I’ve explained in my sequence (and summarized in a gdoc) my distribution has significantly more mass on the 0-6 OOM range than Paul does, and less on the 13+ range. The single post that conveys this intuition most is Fun with +12 OOMs.
Now consider how takeoff speed views interact with timelines views. Paul-slow takeoff and <10 year timelines are in tension with each other. If <7 OOMs of compute would be enough to get something crazy powerful with today’s ideas, then the AI industry is not an efficient market right now. If we get human-level AGI in 2030, then on Paul’s view that means the world economy should be doubling in 2029 and should have doubled over the course of 2025 − 2028 and should already be accelerating now probably. It doesn’t look like that’s happening or about to happen. I think Paul agrees with this; in various conversations he’s said things like “If AGI happens in 10 years or less then probably we get fast takeoff.” [Paul please correct me if I’m mischaracterizing your view!]
Ajeya (and Paul) mostly update against <10 year timelines for this reason. I, by contrast, mostly update against slow takeoff. (Obviously with both do a bit of both, like good Bayesians.)
2. I feel like the debate between EY and Paul (and the broader debate about fast vs. slow takeoff) has been frustratingly much reference class tennis and frustratingly little gears-level modelling. This includes my own writing on the subject—lots of historical analogies and whatnot. I’ve tentatively attempted some things sorta like gears-level modelling (arguably What 2026 Looks Like is an example of this) and so far it seems to be pushing my intuitions more towards “Yep, fast takeoff is more likely.” But I feel like my thinking on this is super inadequate and I think we all should be doing better. Shame! Shame on all of us!
3. I think the focus on GDP (especially GWP) is really off, for reasons mentioned here. I think AI-PONR will probably come before GWP accelerates, and at any rate what we care about for timelines and takeoff speeds is AI-PONR and so our arguments should be about e.g. whether there will be warning shots and powerful AI tools of the sort that are relevant to solving alignment for APS-AI systems.
(Got to go now)
So, there’s this inherent problem with deep gearsy models, where you have to convey a bunch of upstream gears (and the evidence supporting them) before talking about the downstream questions of interest, because if you work backwards then peoples’ brains run out of stack space and they lose track of the whole multi-step path. But if you just go explaining upstream gears first, then people won’t immediately see how they’re relevant to alignment or timelines or whatever, and then lots of people just wander off. Then you go try to explain something about alignment or timelines or whatever, using an argument which relies on those upstream gears, and it goes right over a bunch of peoples’ heads because they don’t have that upstream gear in their world-models.
For the sort of argument in this post, it’s even worse, because a lot of people aren’t even explicitly aware that the relevant type of gear is a thing, or how to think about it beyond a rough intuitive level.
I first ran into this problem in the context of takeoff arguments a couple years ago, and wrote up this sequence mainly to convey the relevant kinds of gears and how to think about them. I claim that this (i.e. constraint slackness/tautness) is usually a good model for gear-type in arguments about reference-classes in practice: typically an intuitively-natural reference class is a set of cases which share some common constraint, and the examples in the reference class then provide evidence for the tautness/slackness of the constraint. For instance, in this post, Paul often points to market efficiency as a taut constraint, and Eliezer argues that constraint is not very taut (at least not in the way needed for the slow takeoff argument). Paul’s intuitive estimates of tautness are presumably driven by things like e.g. financial markets. On the other side, Eliezer wrote Inadequate Equilibria to talk about how taut market efficiency is in general, including gears “further up” and more examples.
If you click through the link in the post to Intelligence Explosion Microeconomics, there’s a lot of this sort of reasoning in it.
The solution might be to start with a concise, low-detail summery (not even one that argues the case, just states it), then start explaining in full detail from the start, knowing that your readers now know which way you’re going.
Wait, I think I just invented the Abstract (not meant as a snide remark. I really did realize it after writing the above, and found it funny).
I didn’t know the last two acronyms despite reading a decent amount of this literature, so thought I’d leave this note for other readers. Listing all of them for completeness (readers will of course know the first two):
TAI: transformative AI
AGI: artificial general intelligence
APS-AI: Advanced, Planning, Strategically aware AI [1]
AI-PONR: AI point of no return [2]
[1] from Carlsmith, which Daniel does link to
[2] from Daniel, which he also linked
Sorry! I’ll go back and insert links + reference your comment