IMO, I don’t particularly like this framing of the player as a (non-socially-constructed)-Goodness-Maximizing AGI”, if only because I view Goodness maximizing as undefined without an arbitrary choice of values, and the implied value misalignment is not what I’d call a goodness maximizer (for the reason that it’s simply unaligned to what the character values).
I like the voice of God/Lovecraftian horror metaphors better.
Goodness maximizing as undefined without an arbitrary choice of values
By “(non-socially-constructed) Goodness” I mean the goodness of a state of affairs as it actually seems to that particular person really-deep-down. Which can have both selfish—perhaps “arbitrary” from a certain perspective—and non-selfish components.
IMO, I don’t particularly like this framing of the player as a (non-socially-constructed)-Goodness-Maximizing AGI”, if only because I view Goodness maximizing as undefined without an arbitrary choice of values, and the implied value misalignment is not what I’d call a goodness maximizer (for the reason that it’s simply unaligned to what the character values).
I like the voice of God/Lovecraftian horror metaphors better.
By “(non-socially-constructed) Goodness” I mean the goodness of a state of affairs as it actually seems to that particular person really-deep-down. Which can have both selfish—perhaps “arbitrary” from a certain perspective—and non-selfish components.
Maybe the crux is I’m more skeptical of the “particular person really deep down” part of ethics.