It is not necessary for the mad AI to kill every single human on the planet. The AI only needs to destroy our ability to respond. In order for human civilization to respond to such a threat we need access to technology, infrastructure and functional social structures (such as the military). The AI could defeat us without killing a single person, if it targets industry and infrastructure. (EY’s melt-all-GPUs nanotech concept could easily be inverted: melt all rubber tires so no one can get to work, for example.)
It is entirely possible that we could see a full Kurzweilian singularity where everything is utopic, and then 20 years later (say) we all drop dead in the same instant. If the AI is both deceptive and patient, it might decide to wait and build up resources, integrate automation (which it can control) into our lives, and manipulate us into thinking everything is okay. Therefore, it is important that we don’t drop our vigilance when we enter a seemingly utopic situation. Ideally we will repeatedly prove to ourselves, and teach every new generation of humans to prove to themselves, that the AI is both values-aligned and corrigible.
Two quick points about DOOM:
It is not necessary for the mad AI to kill every single human on the planet. The AI only needs to destroy our ability to respond. In order for human civilization to respond to such a threat we need access to technology, infrastructure and functional social structures (such as the military). The AI could defeat us without killing a single person, if it targets industry and infrastructure. (EY’s melt-all-GPUs nanotech concept could easily be inverted: melt all rubber tires so no one can get to work, for example.)
It is entirely possible that we could see a full Kurzweilian singularity where everything is utopic, and then 20 years later (say) we all drop dead in the same instant. If the AI is both deceptive and patient, it might decide to wait and build up resources, integrate automation (which it can control) into our lives, and manipulate us into thinking everything is okay. Therefore, it is important that we don’t drop our vigilance when we enter a seemingly utopic situation. Ideally we will repeatedly prove to ourselves, and teach every new generation of humans to prove to themselves, that the AI is both values-aligned and corrigible.
With you on that until
That’s gonna be a lot harder than you think
Unrelated SMBC comic.