I don’t expect an AGI dictator to remain fully psychologically human for an extended period of time (“lifetimes or aeons”), so I wouldn’t bet much on weird quirks of the human psyche saving us all. Like,
I’m hoping that over time their circle would expand through abstract arguments primarily from their interactions with their ASI
This assumes that there would be a “dictator and their ASI” split, as opposed to the dictator merging the ASI’s capabilities into their own mind; this assumes the dictator would decide to retain the quirk of the human mind where humans’ values could be rewritten by external abstract arguments; this assumes that the dictator would keep humans alive long enough for their moral circle to have a chance to expand and for that to do anyone any good; etc.
It also seems to assume that the moral circle would monotonically expand, as opposed to shrinking (e. g., as the result of the dictator growing increasingly callous playing with dolls) or oscillating. I wouldn’t even be confident about that for fully psychologically human people.
I’m definitely not confident in any of this. I do think that these questions aren’t being asked and may wind up being a rather large part of strategy questions on AGI.
If it were me, I’d want to carefully conserve most of how my mind worked while expanding carefully. But I’m not sure what that would mean for how my ethics might change over time.
If it were me, I’d want to carefully conserve most of how my mind worked while expanding carefully
Sure, but if you have an aligned ASI tool at your hands, that still doesn’t preclude radical or rapid changes (at least, rapid on the relevant “incidental expansion of the moral circle” scales). And even if those new types of changes are slow, I think that is still likely to break whatever subtle dynamics may arguably enable gradual moral-circle expansion in humans.
I don’t know. If so, it’s likely to also break any subtle dynamics that contract the moral circle. Right? It seems like it’s moving into territory nobody has thought much about
Yep. But my guess is that it would be a chaotic process, and that outcomes we’d consider acceptable are a narrow target, so on-expectation this would result in a (hyper)existential catastrophe.
I’d expect that for someone with very low priority on being good to other people. Would you expect this also for a fairly good but imperfect person?
I’d at least expect that someone who was pretty good—like most of the nicer folks around here—would take steps to prevent their morality changing enough that their current self would be horrified by their later one. So if they had human flourishing as a fairly high priority, it would probably remain one.
It seems like this would happen if human flourishing were their first priority; if it were second or third to other very different ones, I’d think it less likely to survive intact, but satisficing multiple preferences might preserve a lot of human flourishing if someone wielded that sort of power.
I haven’t been able to find anyone really exploring this logic. I suspect it’s out there somewhere.
I’d expect that for someone with very low priority on being good to other people
Oh, I was assuming we’re talking about that scenario specifically, yes: whether an initially-tyrannical person would end up pro-eudaimonia after lifetimes/aeons of godhood.
I’d at least expect that someone who was pretty good—like most of the nicer folks around here—would take steps to prevent their morality changing enough that their current self would be horrified by their later one
Well, now this becomes a question of competence, not their alignment to human values. If we assume that a good-but-imperfect person who considers human flourishing a high priority ends up in control of an AGI, and that they’re careful enough not to accidentally lose control or self-modify themselves into insanity, then sure, the end result will probably be fine. (Depends on the operationalization of “good but imperfect”, though.)
I haven’t been able to find anyone really exploring this logic
@TsviBT pondered similar questions here, not sure if you saw that?
I don’t expect an AGI dictator to remain fully psychologically human for an extended period of time (“lifetimes or aeons”), so I wouldn’t bet much on weird quirks of the human psyche saving us all. Like,
This assumes that there would be a “dictator and their ASI” split, as opposed to the dictator merging the ASI’s capabilities into their own mind; this assumes the dictator would decide to retain the quirk of the human mind where humans’ values could be rewritten by external abstract arguments; this assumes that the dictator would keep humans alive long enough for their moral circle to have a chance to expand and for that to do anyone any good; etc.
It also seems to assume that the moral circle would monotonically expand, as opposed to shrinking (e. g., as the result of the dictator growing increasingly callous playing with dolls) or oscillating. I wouldn’t even be confident about that for fully psychologically human people.
Interesting.
I’m definitely not confident in any of this. I do think that these questions aren’t being asked and may wind up being a rather large part of strategy questions on AGI.
If it were me, I’d want to carefully conserve most of how my mind worked while expanding carefully. But I’m not sure what that would mean for how my ethics might change over time.
Sure, but if you have an aligned ASI tool at your hands, that still doesn’t preclude radical or rapid changes (at least, rapid on the relevant “incidental expansion of the moral circle” scales). And even if those new types of changes are slow, I think that is still likely to break whatever subtle dynamics may arguably enable gradual moral-circle expansion in humans.
I don’t know. If so, it’s likely to also break any subtle dynamics that contract the moral circle. Right? It seems like it’s moving into territory nobody has thought much about
Yep. But my guess is that it would be a chaotic process, and that outcomes we’d consider acceptable are a narrow target, so on-expectation this would result in a (hyper)existential catastrophe.
I’d expect that for someone with very low priority on being good to other people. Would you expect this also for a fairly good but imperfect person?
I’d at least expect that someone who was pretty good—like most of the nicer folks around here—would take steps to prevent their morality changing enough that their current self would be horrified by their later one. So if they had human flourishing as a fairly high priority, it would probably remain one.
It seems like this would happen if human flourishing were their first priority; if it were second or third to other very different ones, I’d think it less likely to survive intact, but satisficing multiple preferences might preserve a lot of human flourishing if someone wielded that sort of power.
I haven’t been able to find anyone really exploring this logic. I suspect it’s out there somewhere.
Oh, I was assuming we’re talking about that scenario specifically, yes: whether an initially-tyrannical person would end up pro-eudaimonia after lifetimes/aeons of godhood.
Well, now this becomes a question of competence, not their alignment to human values. If we assume that a good-but-imperfect person who considers human flourishing a high priority ends up in control of an AGI, and that they’re careful enough not to accidentally lose control or self-modify themselves into insanity, then sure, the end result will probably be fine. (Depends on the operationalization of “good but imperfect”, though.)
@TsviBT pondered similar questions here, not sure if you saw that?