Even a tiny amount of interest in other people’s well-being can result in huge amounts of that well-being when you don’t have to go to any personal effort, just tell your ASI do it.
This is a complex argument, but I tend to think that anyone with a positive empathy versus sadism balance would tend to become a more generous person over time once they wield unlimited power. It seems like someone with that much power would be prone to understand the situation increasingly better over time, and have to weigh their sadism against their empathy.
It seems clear that empathy and sadism are easily compartmentalized, in that it’s natural to have high empathy for people within the moral circle yet feel sadistically to those outside it. So I don’t think “empathy vs sadism balance” really matters at all, and that the question is almost entirely about where they draw the moral circle. And that seems to be either about their ability to be swayed by abstract arguments, social pressure, and hostility. Since they can control social pressure, that avenue isn’t relevant. It also seems very hard to persuade people to expand the moral circle to people hostile towards them, which is probably a lot of people if you’re an authoritarian. Better hope you get your enlightened philosopher-king! When was the last time we even had one of those, Lee Kwan Yew?
Also, historically, increased power is correlated with increased sadism. People have this weird intuition that becoming more powerful would (at least in many cases) lead to them being more “enlightened”. I think it comes from imagining yourself being more powerful and thus having more affordance to think and care about things that weren’t salient before. I’m sure that would be true for us, but in practice it almost always seems to make people worse.
Right. I’m thinking of a situation with a permanent dictator who has lifetimes or aeons to learn if they aren’t totally unwilling. I’m hoping that over time their circle would expand through abstract arguments primarily from their interactions with their ASI. And I’m very far from sure what types of people would expand their moral circle and which ones just never would.
I wouldn’t call this an intuition. I don’t think people become more enlightened with power; I think they become more enlightened, on average, with knowledge. That’s sharply limited by their fear and resentment, and by their biology; sociopathy is real although it’s probably not a clean zero empathy thing.
I guess the main point is that this is a pretty deep psychological question, and no human in history has been in a position of power as complete and secure as an ASI-enabled dictator would be.
If your point is that this would be pretty terrible in the short term, yes. And the other options look pretty bleak too.
I don’t expect an AGI dictator to remain fully psychologically human for an extended period of time (“lifetimes or aeons”), so I wouldn’t bet much on weird quirks of the human psyche saving us all. Like,
I’m hoping that over time their circle would expand through abstract arguments primarily from their interactions with their ASI
This assumes that there would be a “dictator and their ASI” split, as opposed to the dictator merging the ASI’s capabilities into their own mind; this assumes the dictator would decide to retain the quirk of the human mind where humans’ values could be rewritten by external abstract arguments; this assumes that the dictator would keep humans alive long enough for their moral circle to have a chance to expand and for that to do anyone any good; etc.
It also seems to assume that the moral circle would monotonically expand, as opposed to shrinking (e. g., as the result of the dictator growing increasingly callous playing with dolls) or oscillating. I wouldn’t even be confident about that for fully psychologically human people.
I’m definitely not confident in any of this. I do think that these questions aren’t being asked and may wind up being a rather large part of strategy questions on AGI.
If it were me, I’d want to carefully conserve most of how my mind worked while expanding carefully. But I’m not sure what that would mean for how my ethics might change over time.
If it were me, I’d want to carefully conserve most of how my mind worked while expanding carefully
Sure, but if you have an aligned ASI tool at your hands, that still doesn’t preclude radical or rapid changes (at least, rapid on the relevant “incidental expansion of the moral circle” scales). And even if those new types of changes are slow, I think that is still likely to break whatever subtle dynamics may arguably enable gradual moral-circle expansion in humans.
I don’t know. If so, it’s likely to also break any subtle dynamics that contract the moral circle. Right? It seems like it’s moving into territory nobody has thought much about
Yep. But my guess is that it would be a chaotic process, and that outcomes we’d consider acceptable are a narrow target, so on-expectation this would result in a (hyper)existential catastrophe.
I’d expect that for someone with very low priority on being good to other people. Would you expect this also for a fairly good but imperfect person?
I’d at least expect that someone who was pretty good—like most of the nicer folks around here—would take steps to prevent their morality changing enough that their current self would be horrified by their later one. So if they had human flourishing as a fairly high priority, it would probably remain one.
It seems like this would happen if human flourishing were their first priority; if it were second or third to other very different ones, I’d think it less likely to survive intact, but satisficing multiple preferences might preserve a lot of human flourishing if someone wielded that sort of power.
I haven’t been able to find anyone really exploring this logic. I suspect it’s out there somewhere.
I’d expect that for someone with very low priority on being good to other people
Oh, I was assuming we’re talking about that scenario specifically, yes: whether an initially-tyrannical person would end up pro-eudaimonia after lifetimes/aeons of godhood.
I’d at least expect that someone who was pretty good—like most of the nicer folks around here—would take steps to prevent their morality changing enough that their current self would be horrified by their later one
Well, now this becomes a question of competence, not their alignment to human values. If we assume that a good-but-imperfect person who considers human flourishing a high priority ends up in control of an AGI, and that they’re careful enough not to accidentally lose control or self-modify themselves into insanity, then sure, the end result will probably be fine. (Depends on the operationalization of “good but imperfect”, though.)
I haven’t been able to find anyone really exploring this logic
@TsviBT pondered similar questions here, not sure if you saw that?
I think that’s the Lizardman Constant. I would not be surprised if 10 percent of people replying to a survey pick the strangest or most deranged option just to mess with the surveyor. The fact that this number is the same between the relatively individualist US, the relatively conformist UK, and the relatively fundamentalist Pakistan make me lean towards “let’s mess with this silly grad student” being the common motivation.
Even a tiny amount of interest in other people’s suffering can result in huge amounts of that too. I saw recently that over 10% of people think that if hell does not exist, that we should create it.
It seems clear that empathy and sadism are easily compartmentalized, in that it’s natural to have high empathy for people within the moral circle yet feel sadistically to those outside it. So I don’t think “empathy vs sadism balance” really matters at all, and that the question is almost entirely about where they draw the moral circle. And that seems to be either about their ability to be swayed by abstract arguments, social pressure, and hostility. Since they can control social pressure, that avenue isn’t relevant. It also seems very hard to persuade people to expand the moral circle to people hostile towards them, which is probably a lot of people if you’re an authoritarian. Better hope you get your enlightened philosopher-king! When was the last time we even had one of those, Lee Kwan Yew?
Also, historically, increased power is correlated with increased sadism. People have this weird intuition that becoming more powerful would (at least in many cases) lead to them being more “enlightened”. I think it comes from imagining yourself being more powerful and thus having more affordance to think and care about things that weren’t salient before. I’m sure that would be true for us, but in practice it almost always seems to make people worse.
Right. I’m thinking of a situation with a permanent dictator who has lifetimes or aeons to learn if they aren’t totally unwilling. I’m hoping that over time their circle would expand through abstract arguments primarily from their interactions with their ASI. And I’m very far from sure what types of people would expand their moral circle and which ones just never would.
I wouldn’t call this an intuition. I don’t think people become more enlightened with power; I think they become more enlightened, on average, with knowledge. That’s sharply limited by their fear and resentment, and by their biology; sociopathy is real although it’s probably not a clean zero empathy thing.
I guess the main point is that this is a pretty deep psychological question, and no human in history has been in a position of power as complete and secure as an ASI-enabled dictator would be.
If your point is that this would be pretty terrible in the short term, yes. And the other options look pretty bleak too.
I don’t expect an AGI dictator to remain fully psychologically human for an extended period of time (“lifetimes or aeons”), so I wouldn’t bet much on weird quirks of the human psyche saving us all. Like,
This assumes that there would be a “dictator and their ASI” split, as opposed to the dictator merging the ASI’s capabilities into their own mind; this assumes the dictator would decide to retain the quirk of the human mind where humans’ values could be rewritten by external abstract arguments; this assumes that the dictator would keep humans alive long enough for their moral circle to have a chance to expand and for that to do anyone any good; etc.
It also seems to assume that the moral circle would monotonically expand, as opposed to shrinking (e. g., as the result of the dictator growing increasingly callous playing with dolls) or oscillating. I wouldn’t even be confident about that for fully psychologically human people.
Interesting.
I’m definitely not confident in any of this. I do think that these questions aren’t being asked and may wind up being a rather large part of strategy questions on AGI.
If it were me, I’d want to carefully conserve most of how my mind worked while expanding carefully. But I’m not sure what that would mean for how my ethics might change over time.
Sure, but if you have an aligned ASI tool at your hands, that still doesn’t preclude radical or rapid changes (at least, rapid on the relevant “incidental expansion of the moral circle” scales). And even if those new types of changes are slow, I think that is still likely to break whatever subtle dynamics may arguably enable gradual moral-circle expansion in humans.
I don’t know. If so, it’s likely to also break any subtle dynamics that contract the moral circle. Right? It seems like it’s moving into territory nobody has thought much about
Yep. But my guess is that it would be a chaotic process, and that outcomes we’d consider acceptable are a narrow target, so on-expectation this would result in a (hyper)existential catastrophe.
I’d expect that for someone with very low priority on being good to other people. Would you expect this also for a fairly good but imperfect person?
I’d at least expect that someone who was pretty good—like most of the nicer folks around here—would take steps to prevent their morality changing enough that their current self would be horrified by their later one. So if they had human flourishing as a fairly high priority, it would probably remain one.
It seems like this would happen if human flourishing were their first priority; if it were second or third to other very different ones, I’d think it less likely to survive intact, but satisficing multiple preferences might preserve a lot of human flourishing if someone wielded that sort of power.
I haven’t been able to find anyone really exploring this logic. I suspect it’s out there somewhere.
Oh, I was assuming we’re talking about that scenario specifically, yes: whether an initially-tyrannical person would end up pro-eudaimonia after lifetimes/aeons of godhood.
Well, now this becomes a question of competence, not their alignment to human values. If we assume that a good-but-imperfect person who considers human flourishing a high priority ends up in control of an AGI, and that they’re careful enough not to accidentally lose control or self-modify themselves into insanity, then sure, the end result will probably be fine. (Depends on the operationalization of “good but imperfect”, though.)
@TsviBT pondered similar questions here, not sure if you saw that?
I think that’s the Lizardman Constant. I would not be surprised if 10 percent of people replying to a survey pick the strangest or most deranged option just to mess with the surveyor. The fact that this number is the same between the relatively individualist US, the relatively conformist UK, and the relatively fundamentalist Pakistan make me lean towards “let’s mess with this silly grad student” being the common motivation.