I feel like the debate between EY and Paul (and the broader debate about fast vs. slow takeoff) has been frustratingly much reference class tennis and frustratingly little gears-level modelling.
So, there’s this inherent problem with deep gearsy models, where you have to convey a bunch of upstream gears (and the evidence supporting them) before talking about the downstream questions of interest, because if you work backwards then peoples’ brains run out of stack space and they lose track of the whole multi-step path. But if you just go explaining upstream gears first, then people won’t immediately see how they’re relevant to alignment or timelines or whatever, and then lots of people just wander off. Then you go try to explain something about alignment or timelines or whatever, using an argument which relies on those upstream gears, and it goes right over a bunch of peoples’ heads because they don’t have that upstream gear in their world-models.
For the sort of argument in this post, it’s even worse, because a lot of people aren’t even explicitly aware that the relevant type of gear is a thing, or how to think about it beyond a rough intuitive level.
I first ran into this problem in the context of takeoff arguments a couple years ago, and wrote up this sequence mainly to convey the relevant kinds of gears and how to think about them. I claim that this (i.e. constraint slackness/tautness) is usually a good model for gear-type in arguments about reference-classes in practice: typically an intuitively-natural reference class is a set of cases which share some common constraint, and the examples in the reference class then provide evidence for the tautness/slackness of the constraint. For instance, in this post, Paul often points to market efficiency as a taut constraint, and Eliezer argues that constraint is not very taut (at least not in the way needed for the slow takeoff argument). Paul’s intuitive estimates of tautness are presumably driven by things like e.g. financial markets. On the other side, Eliezer wrote Inadequate Equilibria to talk about how taut market efficiency is in general, including gears “further up” and more examples.
So, there’s this inherent problem with deep gearsy models, where you have to convey a bunch of upstream gears (and the evidence supporting them) before talking about the downstream questions of interest, because if you work backwards then peoples’ brains run out of stack space and they lose track of the whole multi-step path. But if you just go explaining upstream gears first, then people won’t immediately see how they’re relevant to alignment or timelines or whatever, and then lots of people just wander off. Then you go try to explain something about alignment or timelines or whatever, using an argument which relies on those upstream gears, and it goes right over a bunch of peoples’ heads because they don’t have that upstream gear in their world-models.
The solution might be to start with a concise, low-detail summery (not even one that argues the case, just states it), then start explaining in full detail from the start, knowing that your readers now know which way you’re going.
Wait, I think I just invented the Abstract (not meant as a snide remark. I really did realize it after writing the above, and found it funny).
So, there’s this inherent problem with deep gearsy models, where you have to convey a bunch of upstream gears (and the evidence supporting them) before talking about the downstream questions of interest, because if you work backwards then peoples’ brains run out of stack space and they lose track of the whole multi-step path. But if you just go explaining upstream gears first, then people won’t immediately see how they’re relevant to alignment or timelines or whatever, and then lots of people just wander off. Then you go try to explain something about alignment or timelines or whatever, using an argument which relies on those upstream gears, and it goes right over a bunch of peoples’ heads because they don’t have that upstream gear in their world-models.
For the sort of argument in this post, it’s even worse, because a lot of people aren’t even explicitly aware that the relevant type of gear is a thing, or how to think about it beyond a rough intuitive level.
I first ran into this problem in the context of takeoff arguments a couple years ago, and wrote up this sequence mainly to convey the relevant kinds of gears and how to think about them. I claim that this (i.e. constraint slackness/tautness) is usually a good model for gear-type in arguments about reference-classes in practice: typically an intuitively-natural reference class is a set of cases which share some common constraint, and the examples in the reference class then provide evidence for the tautness/slackness of the constraint. For instance, in this post, Paul often points to market efficiency as a taut constraint, and Eliezer argues that constraint is not very taut (at least not in the way needed for the slow takeoff argument). Paul’s intuitive estimates of tautness are presumably driven by things like e.g. financial markets. On the other side, Eliezer wrote Inadequate Equilibria to talk about how taut market efficiency is in general, including gears “further up” and more examples.
If you click through the link in the post to Intelligence Explosion Microeconomics, there’s a lot of this sort of reasoning in it.
The solution might be to start with a concise, low-detail summery (not even one that argues the case, just states it), then start explaining in full detail from the start, knowing that your readers now know which way you’re going.
Wait, I think I just invented the Abstract (not meant as a snide remark. I really did realize it after writing the above, and found it funny).