My P(doom) is 100% - Ɛ to Ɛ: 100% - Ɛ, Humanity has already destroyed the solar system (and the AGI is rapidly expanding to the rest of the galaxy) on at least some of the quantum branches due to an independence gaining AGI accident. I am fairly certain this was possible by 2010, and possibly as soon as around 1985. Ɛ, at least some of the quantum branches where independence gaining AGI happens will have a sufficiently benign AGI that we survive the experience or we will actually restrict computers enough to stop accidentally creating AGI. It is worth noting that if AGI has a very high probability of killing people, what it looks like in a world with quantum branching is periodically there will be AI winters when computers and techniques approach being capable of AGI, because many of the “successful” uses of AI result in deadly AGI, and so we just don’t live long enough to observe those.
And if I am just talking with an average person, I say > 1%, and make the side comment that if a passenger plane had a 1% probability of crashing in the next flight, it would not take off.
Edit: And to really explain this would take a lot longer post, so I apologize for that. That said, if I had to choose, to prevent accidental AGI, I would be willing to restrict myself and everyone else to computers on the order of 512 KiB of RAM, 2 MiB of disk space as part of a comprehensive program to prevent existential risk.
My P(doom) is 100% - Ɛ to Ɛ: 100% - Ɛ, Humanity has already destroyed the solar system (and the AGI is rapidly expanding to the rest of the galaxy) on at least some of the quantum branches due to an independence gaining AGI accident. I am fairly certain this was possible by 2010, and possibly as soon as around 1985. Ɛ, at least some of the quantum branches where independence gaining AGI happens will have a sufficiently benign AGI that we survive the experience or we will actually restrict computers enough to stop accidentally creating AGI. It is worth noting that if AGI has a very high probability of killing people, what it looks like in a world with quantum branching is periodically there will be AI winters when computers and techniques approach being capable of AGI, because many of the “successful” uses of AI result in deadly AGI, and so we just don’t live long enough to observe those.
And if I am just talking with an average person, I say > 1%, and make the side comment that if a passenger plane had a 1% probability of crashing in the next flight, it would not take off.
Edit: And to really explain this would take a lot longer post, so I apologize for that. That said, if I had to choose, to prevent accidental AGI, I would be willing to restrict myself and everyone else to computers on the order of 512 KiB of RAM, 2 MiB of disk space as part of a comprehensive program to prevent existential risk.