My impression from the story is not that the individual people (moms, dads, little girls, etc) will individually be converted to super-happy individuals (such as portrayed in Yudkowsky’s story “Three Worlds Collide”), but rather their bodies will be torn apart (alongside the rest of the solar system) to be used for the construction of myriad tiny computers (or perhaps some biological substrate, like some kind of maximum-happiness algae) repeatedly computing some basic circuit representing maximum bliss. Or, more optimistically in my view, maybe instead of a bunch of repetitive tiny calculations, it’s one big super-complex calculation of maximum bliss, like the whole world is getting eaten by a very happy god. But in either case, people’s individual identities, memories, stories, etc, are going to get torn apart and destroyed. IMO this drives the central tension of the story—the little girl is excitedly wondering what life will be like when she goes to happy-heaven, but the adults know that no existing person will have a future after 10 days from now; they are all simply going to get eaten by a happy god / happy algae / etc. The atoms in their bodies will be repurposed into something very happy, but there’s no sense in which they themselves will experience the hedonium era.
I think longtermist utilitarians were making this a real concern long before AI.
(To be clear I actually find this moral view potentially plausible, although I’m pretty morally uncertain. I think it would be pretty surprising and definitely a mistake a if we intentionally don’t give current (post-)humans at least the Milky Way Galaxy, but it’s possible that it could be good to tile the rest of the universe with whatever we decide/figure out is morally optimal.)
Ah I see, good point. Well, in that world, setting aside the epistemic question of how could we know it’s actually a positive valence hedonium shockwave, if I had good reason to believe it was a true hedonium shockwave I would still be overall quite happy about it. It would be much more bittersweet than my original interpretation, and now I understand much better why the adults of the story are upset, but… idk, there’s so much intense suffering on earth that I’d rather have hedonium even if it means that “me” doesn’t go on.
Going back to the epistemic question, I think it would be impossible to know that the shockwave is hedonium with our current understanding of consciousness. So if all I can properly know is we’re all gonna get wiped out by some kind of tiling mechanism or superentity, then it goes back to being extremely sad (with the elimination of extreme suffering on earth as a silver lining, except that it could also be an antihedonium shockwave which would be far worse).
I know that’s a controversial take, if it makes you feel better I don’t think anyone should initiate a hedonium shockwave because it’s impossible to know that it will actually work, and so Eliezers reasoning about why the ends don’t justify the means applies (“for the good of the tribe don’t do what’s best for the tribe”). We are way too limited epistemologically (biases, blindspots, etc) to do it right, and the problem requires so much care that I don’t think any mind, even a superintelligent ai or alien, would be able to overcome that issue.
My impression from the story is not that the individual people (moms, dads, little girls, etc) will individually be converted to super-happy individuals (such as portrayed in Yudkowsky’s story “Three Worlds Collide”), but rather their bodies will be torn apart (alongside the rest of the solar system) to be used for the construction of myriad tiny computers (or perhaps some biological substrate, like some kind of maximum-happiness algae) repeatedly computing some basic circuit representing maximum bliss. Or, more optimistically in my view, maybe instead of a bunch of repetitive tiny calculations, it’s one big super-complex calculation of maximum bliss, like the whole world is getting eaten by a very happy god. But in either case, people’s individual identities, memories, stories, etc, are going to get torn apart and destroyed. IMO this drives the central tension of the story—the little girl is excitedly wondering what life will be like when she goes to happy-heaven, but the adults know that no existing person will have a future after 10 days from now; they are all simply going to get eaten by a happy god / happy algae / etc. The atoms in their bodies will be repurposed into something very happy, but there’s no sense in which they themselves will experience the hedonium era.
Funny[1] how AI can make the whole “utility monster” critique of utilitarianism a real, fundamental concern instead of a handwaving hypothetical.
And very concerning.
I think longtermist utilitarians were making this a real concern long before AI.
(To be clear I actually find this moral view potentially plausible, although I’m pretty morally uncertain. I think it would be pretty surprising and definitely a mistake a if we intentionally don’t give current (post-)humans at least the Milky Way Galaxy, but it’s possible that it could be good to tile the rest of the universe with whatever we decide/figure out is morally optimal.)
Surely you mean experiencing maximum bliss. That is completely different from merely “representing” it.
“Representing” maximum bliss would be like writing “I love school” 100 times on the blackboard. That does not make me actually love school...
Ah I see, good point. Well, in that world, setting aside the epistemic question of how could we know it’s actually a positive valence hedonium shockwave, if I had good reason to believe it was a true hedonium shockwave I would still be overall quite happy about it. It would be much more bittersweet than my original interpretation, and now I understand much better why the adults of the story are upset, but… idk, there’s so much intense suffering on earth that I’d rather have hedonium even if it means that “me” doesn’t go on.
Going back to the epistemic question, I think it would be impossible to know that the shockwave is hedonium with our current understanding of consciousness. So if all I can properly know is we’re all gonna get wiped out by some kind of tiling mechanism or superentity, then it goes back to being extremely sad (with the elimination of extreme suffering on earth as a silver lining, except that it could also be an antihedonium shockwave which would be far worse).
I know that’s a controversial take, if it makes you feel better I don’t think anyone should initiate a hedonium shockwave because it’s impossible to know that it will actually work, and so Eliezers reasoning about why the ends don’t justify the means applies (“for the good of the tribe don’t do what’s best for the tribe”). We are way too limited epistemologically (biases, blindspots, etc) to do it right, and the problem requires so much care that I don’t think any mind, even a superintelligent ai or alien, would be able to overcome that issue.