I have the sense that rationalists think there’s a a very important distinction between “literally everyone will die” and, say, “the majority of people will suffer and/or die.” I do not share that sense, and to me, the burden of proof set by the title is unreasonably high.
The distinction is human endeavor continuing vs. not. Though survival of some or even essentially all humans doesn’t necessarily mean that the human endeavor survives without being permanently crippled. The AIs might leave only a tiny sliver of the future resources for the future of humanity, with no prospect at all of this ever changing, even on cosmic timescales (permanent disempowerment). The IABIED thesis is that even this is very unlikely, but it’s a controversial point. And the transition to this regime doesn’t necessarily involve an explicit takeover, as humanity voluntarily hands off influence to AIs, more and more of it, without bound (gradual disempowerment).
So I expect that if there are survivors after “the majority of people will suffer and/or die”, that’s either a human-initiated catastrophe (misuse of AI), or an instrumentally motivated AI takeover (when it’s urgent for the AIs to stop whatever humanity would be doing at that time if left intact) that transitions to either complete extinction or permanent disempowerment that offers no prospect ever of a true recovery (depending on whether AIs still terminally value preserving human life a little bit, even if regrettably they couldn’t afford to do so perfectly).
Permanent disempowerment leaves humanity completely at the mercy of AIs (even if we got there through gradual disempowerment, possibly with no takeover at all). It implies that the ultimate outcome is fully determined by values of AIs, and the IABIED arguments seem strong enough for at least some significant probability that the AIs in charge will end up with zero mercy (the IABIED authors believe that their arguments should carry this even further, making it very likely instead).
The distinction is human endeavor continuing vs. not. Though survival of some or even essentially all humans doesn’t necessarily mean that the human endeavor survives without being permanently crippled. The AIs might leave only a tiny sliver of the future resources for the future of humanity, with no prospect at all of this ever changing, even on cosmic timescales (permanent disempowerment). The IABIED thesis is that even this is very unlikely, but it’s a controversial point. And the transition to this regime doesn’t necessarily involve an explicit takeover, as humanity voluntarily hands off influence to AIs, more and more of it, without bound (gradual disempowerment).
So I expect that if there are survivors after “the majority of people will suffer and/or die”, that’s either a human-initiated catastrophe (misuse of AI), or an instrumentally motivated AI takeover (when it’s urgent for the AIs to stop whatever humanity would be doing at that time if left intact) that transitions to either complete extinction or permanent disempowerment that offers no prospect ever of a true recovery (depending on whether AIs still terminally value preserving human life a little bit, even if regrettably they couldn’t afford to do so perfectly).
Permanent disempowerment leaves humanity completely at the mercy of AIs (even if we got there through gradual disempowerment, possibly with no takeover at all). It implies that the ultimate outcome is fully determined by values of AIs, and the IABIED arguments seem strong enough for at least some significant probability that the AIs in charge will end up with zero mercy (the IABIED authors believe that their arguments should carry this even further, making it very likely instead).