Did the book convince you that if superintelligence is built in the next 20 years (however that happens, if it does, and for at least some sensible threshold-like meaning of “superintelligence”), then there is at least a 5-10% chance that as a result literally everyone will literally die?
I’m much more in the world of Knightian uncertainty here (i.e., it could happen but I have no idea how to quantify that) than in one where I feel like I can reasonably collapse it into a clear, probabilistic risk. I am persuaded that this is something that cannot be ruled out.
I have the sense that rationalists think there’s a a very important distinction between “literally everyone will die” and, say, “the majority of people will suffer and/or die.” I do not share that sense, and to me, the burden of proof set by the title is unreasonably high.
I’ll assent to the statement that there’s at least a 10% chance of something very bad happening, where “very bad” means >50% of people dying or experiencing severe suffering or something equivalent to that.
I think this kind of claim is the crux for motivating some sort of global ban or pause on rushed advanced general AI development in the near future (as an input to policy separate from the difficulty of actually making this happen). Or for not being glad that there is an “AI race” (even if it’s very hard to mitigate). So it’s interesting if your “not sure on existential risk” takeaway is denying or affirming this claim.
Give me a magic, zero-side effect pause button, and I’ll hit it instantly.
I have the sense that rationalists think there’s a a very important distinction between “literally everyone will die” and, say, “the majority of people will suffer and/or die.” I do not share that sense, and to me, the burden of proof set by the title is unreasonably high.
The distinction is human endeavor continuing vs. not. Though survival of some or even essentially all humans doesn’t necessarily mean that the human endeavor survives without being permanently crippled. The AIs might leave only a tiny sliver of the future resources for the future of humanity, with no prospect at all of this ever changing, even on cosmic timescales (permanent disempowerment). The IABIED thesis is that even this is very unlikely, but it’s a controversial point. And the transition to this regime doesn’t necessarily involve an explicit takeover, as humanity voluntarily hands off influence to AIs, more and more of it, without bound (gradual disempowerment).
So I expect that if there are survivors after “the majority of people will suffer and/or die”, that’s either a human-initiated catastrophe (misuse of AI), or an instrumentally motivated AI takeover (when it’s urgent for the AIs to stop whatever humanity would be doing at that time if left intact) that transitions to either complete extinction or permanent disempowerment that offers no prospect ever of a true recovery (depending on whether AIs still terminally value preserving human life a little bit, even if regrettably they couldn’t afford to do so perfectly).
Permanent disempowerment leaves humanity completely at the mercy of AIs (even if we got there through gradual disempowerment, possibly with no takeover at all). It implies that the ultimate outcome is fully determined by values of AIs, and the IABIED arguments seem strong enough for at least some significant probability that the AIs in charge will end up with zero mercy (the IABIED authors believe that their arguments should carry this even further, making it very likely instead).
I have the sense that rationalists think there’s a a very important distinction between “literally everyone will die” and, say, “the majority of people will suffer and/or die.” I do not share that sense, and to me, the burden of proof set by the title is unreasonably high.
I would say that there is a distinction, but I agree that at those level of badness it sort of blurs out in a single blob of awfulness. But generally speaking I see it as, if someone was told “your whole family will be killed except your youngest son” or “your whole family will be killed, no one survives”… obviously both scenarios are horrifying but still you’d marginally prefer the first one. I think if people fall in the trap of being so taken by the extinction risk that they brush off a scenario in which, say, 95% of all people die, then they’re obviously losing perspective, but I also think it’s fair to say that the loss of all of humanity is worse than just the sum total of the loss of each individual in it (same reason why we consider genocide bad in and of its own—it’s not just the loss of people, it’s the loss of culture, knowledge, memory, on top of the people).
I’m much more in the world of Knightian uncertainty here (i.e., it could happen but I have no idea how to quantify that) than in one where I feel like I can reasonably collapse it into a clear, probabilistic risk. I am persuaded that this is something that cannot be ruled out.
I have the sense that rationalists think there’s a a very important distinction between “literally everyone will die” and, say, “the majority of people will suffer and/or die.” I do not share that sense, and to me, the burden of proof set by the title is unreasonably high.
I’ll assent to the statement that there’s at least a 10% chance of something very bad happening, where “very bad” means >50% of people dying or experiencing severe suffering or something equivalent to that.
Give me a magic, zero-side effect pause button, and I’ll hit it instantly.
The distinction is human endeavor continuing vs. not. Though survival of some or even essentially all humans doesn’t necessarily mean that the human endeavor survives without being permanently crippled. The AIs might leave only a tiny sliver of the future resources for the future of humanity, with no prospect at all of this ever changing, even on cosmic timescales (permanent disempowerment). The IABIED thesis is that even this is very unlikely, but it’s a controversial point. And the transition to this regime doesn’t necessarily involve an explicit takeover, as humanity voluntarily hands off influence to AIs, more and more of it, without bound (gradual disempowerment).
So I expect that if there are survivors after “the majority of people will suffer and/or die”, that’s either a human-initiated catastrophe (misuse of AI), or an instrumentally motivated AI takeover (when it’s urgent for the AIs to stop whatever humanity would be doing at that time if left intact) that transitions to either complete extinction or permanent disempowerment that offers no prospect ever of a true recovery (depending on whether AIs still terminally value preserving human life a little bit, even if regrettably they couldn’t afford to do so perfectly).
Permanent disempowerment leaves humanity completely at the mercy of AIs (even if we got there through gradual disempowerment, possibly with no takeover at all). It implies that the ultimate outcome is fully determined by values of AIs, and the IABIED arguments seem strong enough for at least some significant probability that the AIs in charge will end up with zero mercy (the IABIED authors believe that their arguments should carry this even further, making it very likely instead).
I would say that there is a distinction, but I agree that at those level of badness it sort of blurs out in a single blob of awfulness. But generally speaking I see it as, if someone was told “your whole family will be killed except your youngest son” or “your whole family will be killed, no one survives”… obviously both scenarios are horrifying but still you’d marginally prefer the first one. I think if people fall in the trap of being so taken by the extinction risk that they brush off a scenario in which, say, 95% of all people die, then they’re obviously losing perspective, but I also think it’s fair to say that the loss of all of humanity is worse than just the sum total of the loss of each individual in it (same reason why we consider genocide bad in and of its own—it’s not just the loss of people, it’s the loss of culture, knowledge, memory, on top of the people).