By default, this means that people should mostly ignore the worst or best outcomes if they want to predict how a crisis or x-risk will go. This has important implications for how bad things are likely to go in slow takeoff.
What if the median outcome and the worst outcome are equivalent, in that humanity dies in both?
Shouldn’t we be aiming for—at very least—the median outcome in which humanity survives? Even if that is only a small part of the total space of outcomes?
Perhaps the world really is such that we have a 1% chance that we fully solve alignment and live in a glorious transhuman utopia, a 48% chance that we end up in a good world with partly aligned AGI, a 1% chance that we coordinate on never developing AGI, a 49% chance that we end up in a bad world with partly unaligned AGI, and a 1% chance that we all die screaming from completely unaligned AGI.
But maybe the world really is such that there is a 1% chance that we fully solve alignment and live in a transhuman utopia, a 1% chance that we coordinate on never developing AGI, a 1% chance that we partly solve alignment and end up in a permanent dystopia that is nevertheless better than extinction, and a 97% chance that UFAI kills us all or worse.
We don’t know which way the world really is, and focusing on “the median scenario” won’t help at all with that. My estimate is that the real world is much closer to the second than to the first, and I see no point in focusing on the “median scenario” in that.
What if the median outcome and the worst outcome are equivalent, in that humanity dies in both?
Shouldn’t we be aiming for—at very least—the median outcome in which humanity survives? Even if that is only a small part of the total space of outcomes?
Perhaps the world really is such that we have a 1% chance that we fully solve alignment and live in a glorious transhuman utopia, a 48% chance that we end up in a good world with partly aligned AGI, a 1% chance that we coordinate on never developing AGI, a 49% chance that we end up in a bad world with partly unaligned AGI, and a 1% chance that we all die screaming from completely unaligned AGI.
But maybe the world really is such that there is a 1% chance that we fully solve alignment and live in a transhuman utopia, a 1% chance that we coordinate on never developing AGI, a 1% chance that we partly solve alignment and end up in a permanent dystopia that is nevertheless better than extinction, and a 97% chance that UFAI kills us all or worse.
We don’t know which way the world really is, and focusing on “the median scenario” won’t help at all with that. My estimate is that the real world is much closer to the second than to the first, and I see no point in focusing on the “median scenario” in that.