I’m retreating from my previous argument a bit. The AGI doesn’t need to cause literal human extinction with a virus; if it can cause enough damage to collapse human industrial civilization (while being able to survive said collapse) then that would also achieve most of the AGI’s goal of being able to do what it wants without humans stopping it. Naturally occurring pathogens from Europe devastated Native American populations after Columbus; throw a bunch of bad enough novel viruses at us at once and you probably could knock humanity back to the metaphorical Stone Age.
I find that more plausible. Also horrifying and worth fighting against, but not what EY is saying
Note that EY is saying “there exists a real plan that is at least as dangerous as this one”; if you think there is such a plan, then you can agree with the conclusion, even if you don’t agree with his example. [There is an epistemic risk here, if everyone mistakenly believes that a different doomsday plan is possible when someone else knows why that specific plan won’t work, and so if everyone pooled all their knowledge they could know that none of the plans will work. But I’m moderately confident we’re instead in a world with enough vulnerabilities that broadcasting them makes things worse instead of better.]
I’m retreating from my previous argument a bit. The AGI doesn’t need to cause literal human extinction with a virus; if it can cause enough damage to collapse human industrial civilization (while being able to survive said collapse) then that would also achieve most of the AGI’s goal of being able to do what it wants without humans stopping it. Naturally occurring pathogens from Europe devastated Native American populations after Columbus; throw a bunch of bad enough novel viruses at us at once and you probably could knock humanity back to the metaphorical Stone Age.
I find that more plausible. Also horrifying and worth fighting against, but not what EY is saying
Note that EY is saying “there exists a real plan that is at least as dangerous as this one”; if you think there is such a plan, then you can agree with the conclusion, even if you don’t agree with his example. [There is an epistemic risk here, if everyone mistakenly believes that a different doomsday plan is possible when someone else knows why that specific plan won’t work, and so if everyone pooled all their knowledge they could know that none of the plans will work. But I’m moderately confident we’re instead in a world with enough vulnerabilities that broadcasting them makes things worse instead of better.]