It feels like there’s a huge blind spot in this post, and it saddens (and scares) me to say it. The possible outcomes are not utopia for billions of years or bust. The possible outcomes are utopia for billions of years, distopia for billions of years, or bust. Without getting into the details, I can imagine S-tier risks in which the AGI turns out to care too much about engagement from alive humans, and things getting dark from there.
Short of pretty much torture for eternity, the “keep humans around but drug them to increase their happiness” scenarios are also distopian and may also be worse than death. Are there good reasons to expect utopia is more likely relative to distopian (with extinction remaining most likely)?
Yes, exactly. C.S. Lewis wrote a very weird science fiction book titled That Hideous Strength, that was about (basically) a biological version of the Singularity.
And there’s a scene where one the villains is explaining that with immortality, it will finally be possible to damn people to eternal Hell.
And of course, “Hells” are a significant theme in at least one of Iain M Banks’ Culture novels as well.
This is a very obvious corollary: If there exists an entity powerful enough to build an immortal utopia, there is necessarily an entity powerful enough to inflict eternal suffering. It’s unclear whether humans could ever control such a thing. And even if we could, that would also mean that some humans in particular would control the AI. How many AI lab CEOs would you trust with the power of eternal damnation?
(This is one of several reasons why I support an AI halt. I do not think that power should exist, no matter who or what controls it.)
Getting AI to terminally care about humans at all seems like a hard target and if our alignment efforts can make it happen, they can probably also ensure that it cares about humans in a good way.
It feels like there’s a huge blind spot in this post, and it saddens (and scares) me to say it. The possible outcomes are not utopia for billions of years or bust. The possible outcomes are utopia for billions of years, distopia for billions of years, or bust. Without getting into the details, I can imagine S-tier risks in which the AGI turns out to care too much about engagement from alive humans, and things getting dark from there.
Short of pretty much torture for eternity, the “keep humans around but drug them to increase their happiness” scenarios are also distopian and may also be worse than death. Are there good reasons to expect utopia is more likely relative to distopian (with extinction remaining most likely)?
To summarize this risk eloquently, if we build God, we build the real possibility of Hell.
Yes, exactly. C.S. Lewis wrote a very weird science fiction book titled That Hideous Strength, that was about (basically) a biological version of the Singularity.
And there’s a scene where one the villains is explaining that with immortality, it will finally be possible to damn people to eternal Hell.
And of course, “Hells” are a significant theme in at least one of Iain M Banks’ Culture novels as well.
This is a very obvious corollary: If there exists an entity powerful enough to build an immortal utopia, there is necessarily an entity powerful enough to inflict eternal suffering. It’s unclear whether humans could ever control such a thing. And even if we could, that would also mean that some humans in particular would control the AI. How many AI lab CEOs would you trust with the power of eternal damnation?
(This is one of several reasons why I support an AI halt. I do not think that power should exist, no matter who or what controls it.)
Getting AI to terminally care about humans at all seems like a hard target and if our alignment efforts can make it happen, they can probably also ensure that it cares about humans in a good way.
Current LLMs could probably be said to care about humans in some way, but I’d be pretty scared to live in an LLM dictatorship.