Curious. Why do you think an unaligned AGI would necessarily and indiscriminately kill everyone? Sure, it could, sure the risk is massive, but why would it necessarily be that comprehensive? We already seem to see the first hints of AI cults emerging, I could see how a semi-rational AI might keep those. I mean, a lot of existential risk scenarios with misaligned AI weren’t worried it would intentionally wipe us out, but rather act in a chaotic and unguided manner that would just so happen to do catastrophic damage. I don’t think we can predict what an unaligned AI would do. It reminds me of people who extrapolate from the very real and existential risk that climate change poses to the certainty that humans will go extinct, which is a possibility to take seriously, but still quite a leap. We might just be reduced to stone age numbers and civilisation level.
Curious. Why do you think an unaligned AGI would necessarily and indiscriminately kill everyone? Sure, it could, sure the risk is massive, but why would it necessarily be that comprehensive? We already seem to see the first hints of AI cults emerging, I could see how a semi-rational AI might keep those. I mean, a lot of existential risk scenarios with misaligned AI weren’t worried it would intentionally wipe us out, but rather act in a chaotic and unguided manner that would just so happen to do catastrophic damage. I don’t think we can predict what an unaligned AI would do. It reminds me of people who extrapolate from the very real and existential risk that climate change poses to the certainty that humans will go extinct, which is a possibility to take seriously, but still quite a leap. We might just be reduced to stone age numbers and civilisation level.
Maybe this podcast with Eliezer could help explain that.