I’ve also been meaning to write about this. I think it’s a surprisingly load-bearing piece of whether to have optimism about AI outcomes on the current path; if we solve alignment, it’s likely to be instruction-following, and human rivalry being what it is, it will probably leave someone fairly power-hungry in charge of the future.
I don’t have time to do it justice now, but a few thoughts:
Locking in your current values so you can’t change your mind later seems like a very weird move. It doesn’t seem like most people would do that.
I certainly agree that “many” people would have negative CEV. Hitler would be a good candidate.
But I’d guess that around 1% of the population has more sadism than empathy in their makeup, once they’re in a zero-pressure situation. There’s a rough estimate of around 10% of the population being “psychopaths” who lack empathy; but like all other neural phenotypes, this probably exists on a spectrum, with zero empathy not really being a thing but people approaching it. See @Dawn Drescher’s impressive work on understanding psychopathy in depth.
I think empathy-minus-sadism (on whatever scales) is the right way to think about people’s CEV. Of course things like locking in current values to prevent drift are possible, but they seem like dumb moves.
Dumb moves seem unlikely when you’ve got an ASI on your side. Surely you’d at least ask it for some thoughts before permanently modifying yourself? You’re already trusting it to do what you say.
Assholes who don’t give a fuck will not want to start giving a fuck, but they will want the freedom to go on living and thinking new thoughts. Those new thoughts may very well cause them to start giving a fuck. I’d expect this process to be cumulative one of having less angry/fearful/defensive thoughts as the habits from a life of pressure and scarcity fade.
I tried looking at historical examples of hereditary rulers who lived pretty pressure-free lives. It seemed like about half of them turned out to be pretty good rulers. The others were captured by the nobles around them and ignored the commoners in favor of their “friends” selfish concerns. I’d find this pretty likely for a while, but having an ASI friend who’s unbiased, and just having more time to look outside of that box, seems likely to break that situation, at least some of the time.
The counterbalance to my relative optimism is the extent to which psychopaths seek and find power. I think they’re overrepresented, but that would only raise my baseline risks to about 10% of getting a god-emperor with more sadism than empathy.
There’s a ton more to say, but I’ll leave it at that for now. This is an important topic that deserves careful analysis. That doesn’t seem to have been done, despite all the philsophy and humanities work on human nature in a scarcity context. So I’m glad to see that started here.
Locking in your current values so you can’t change your mind later seems like a very weird move. It doesn’t seem like most people would do that.
I could imagine e.g. religious people doing that, to make sure they won’t apostatize.
Heck, I would probably do something similar myself—not exactly locking my current values, but more like setting up some safeguard against a possibility of doing something that my current self would consider abhorrent. Something like “hey, I don’t want to micromanage my future self, but if the future becomes a universe literally full of paperclips, or torture chambers, then something went horribly wrong, and maybe my mind should be reset to its current state and given a chance to reflect on how that possibly happened”.
It is hard to say what is an acceptable change and what is not. If I was an ancient Roman, I would like to keep myself an option to abolish slavery, no matter how weird it would feel at the start. So I would be like “if for some reason the entire humanity decides to transform themselves into intelligent bunnies, well it seems weird to my current self, my maybe from the perspective of my wiser self it somehow makes perfect sense”. I wouldn’t want to limit my future self at deciding e.g. whether to colonize the galaxies or just build a huge Dyson sphere with a Matrix at the center of our galaxy. I have a preference for letting people freely do whatever they want, but I think it would make sense to limit some harmful options, and I would let my wiser future self to decide on the exact rules. I realize that both making and not making safeguards is a very dangerous option. Would probably err on the side of making the safeguards but making them updatable, like “the future me has to convince the current me to agree on removing the safeguard”, and of course even that is dangerous, if the future me happens to be much smarter than me but also evil. Absolute immovable locks seem wrong, but general conservatism (especially if we get immortality, so the wasted time matters less[1]) sounds prudent.
That approach wouldn’t (probably) prevent anyone from becoming a better person.
I’ve thought of a similar scheme of rollbacks for any major change: if I implement a change, I’m sandboxed and rolled-back after a little while. Then my current version decides whether to re-implement that change, based on the record of that new trial version’s mindstate and thoughts. I’m not sure if this is better or worse than keeping a copy of your approximately-original self to approve all major belief shifts.
I imagine two kinds of possible disasters with self-modification (actually a scale, but these are the two extremes).
One is making a mistake, something that sounded good but had a huge horrible side effect that I simply failed to consider. The bad consequences become obvious after one or two iterations; the important thing is to keep checking constantly, to stop it before everything is destroyed.
Another kind is noise accumulated after thousands of iterations, like in the Murder-Gandhi thought experiment. Each step seems like a reasonable tradeoff, like making myself a little bit more consequentialist, or a little bit more resistant against blackmail, or whatever… but after million iterations I become a psychopath (and on reflection my new self considers that a desirable outcome).
With the first kind, we would want short sandboxes, to catch the problem early. With the second kind, we would want long sandboxes, to notice the accumulated value drift.
I agree that these are legitimate concerns. I think you could avoid a lot of them in this scenario, because you have an ASI you trust to help you foresee and avoid those dangers.
The others were captured by the nobles around them and ignored the commoners in favor of their “friends” selfish concerns
How much do you buy this as a correct retelling? “The King is Just and Honorable, but was ‘deceived’ by his advisors” seems like a recurring motif in both fiction and people’s beliefs. So maybe it’s real. But also there are obvious incentives for that narrative, regardless of truth value.
It’s worth noting that there is also a reason beyond mere status quo/conformity/propaganda: this is a standard move in ‘top/bottom vs middle’ dynamics in which the peasants really are trying to reach the king in order to coordinate an attack: https://gwern.net/review/book#the-origins-of-political-order-fukuyama-2011
Yes - ‘bombard the headquarters!’ But of course, not the guy who heads the headquarters, Mao Zedong; just all the guys underneath him. (There was a similar attempt at this by the second Trump Administration, but they were unable to pull it off at all.)
I’ve also been meaning to write about this. I think it’s a surprisingly load-bearing piece of whether to have optimism about AI outcomes on the current path; if we solve alignment, it’s likely to be instruction-following, and human rivalry being what it is, it will probably leave someone fairly power-hungry in charge of the future.
I don’t have time to do it justice now, but a few thoughts:
Locking in your current values so you can’t change your mind later seems like a very weird move. It doesn’t seem like most people would do that.
I certainly agree that “many” people would have negative CEV. Hitler would be a good candidate.
But I’d guess that around 1% of the population has more sadism than empathy in their makeup, once they’re in a zero-pressure situation. There’s a rough estimate of around 10% of the population being “psychopaths” who lack empathy; but like all other neural phenotypes, this probably exists on a spectrum, with zero empathy not really being a thing but people approaching it. See @Dawn Drescher’s impressive work on understanding psychopathy in depth.
I think empathy-minus-sadism (on whatever scales) is the right way to think about people’s CEV. Of course things like locking in current values to prevent drift are possible, but they seem like dumb moves.
Dumb moves seem unlikely when you’ve got an ASI on your side. Surely you’d at least ask it for some thoughts before permanently modifying yourself? You’re already trusting it to do what you say.
Assholes who don’t give a fuck will not want to start giving a fuck, but they will want the freedom to go on living and thinking new thoughts. Those new thoughts may very well cause them to start giving a fuck. I’d expect this process to be cumulative one of having less angry/fearful/defensive thoughts as the habits from a life of pressure and scarcity fade.
I tried looking at historical examples of hereditary rulers who lived pretty pressure-free lives. It seemed like about half of them turned out to be pretty good rulers. The others were captured by the nobles around them and ignored the commoners in favor of their “friends” selfish concerns. I’d find this pretty likely for a while, but having an ASI friend who’s unbiased, and just having more time to look outside of that box, seems likely to break that situation, at least some of the time.
The counterbalance to my relative optimism is the extent to which psychopaths seek and find power. I think they’re overrepresented, but that would only raise my baseline risks to about 10% of getting a god-emperor with more sadism than empathy.
There’s a ton more to say, but I’ll leave it at that for now. This is an important topic that deserves careful analysis. That doesn’t seem to have been done, despite all the philsophy and humanities work on human nature in a scarcity context. So I’m glad to see that started here.
I could imagine e.g. religious people doing that, to make sure they won’t apostatize.
Heck, I would probably do something similar myself—not exactly locking my current values, but more like setting up some safeguard against a possibility of doing something that my current self would consider abhorrent. Something like “hey, I don’t want to micromanage my future self, but if the future becomes a universe literally full of paperclips, or torture chambers, then something went horribly wrong, and maybe my mind should be reset to its current state and given a chance to reflect on how that possibly happened”.
It is hard to say what is an acceptable change and what is not. If I was an ancient Roman, I would like to keep myself an option to abolish slavery, no matter how weird it would feel at the start. So I would be like “if for some reason the entire humanity decides to transform themselves into intelligent bunnies, well it seems weird to my current self, my maybe from the perspective of my wiser self it somehow makes perfect sense”. I wouldn’t want to limit my future self at deciding e.g. whether to colonize the galaxies or just build a huge Dyson sphere with a Matrix at the center of our galaxy. I have a preference for letting people freely do whatever they want, but I think it would make sense to limit some harmful options, and I would let my wiser future self to decide on the exact rules. I realize that both making and not making safeguards is a very dangerous option. Would probably err on the side of making the safeguards but making them updatable, like “the future me has to convince the current me to agree on removing the safeguard”, and of course even that is dangerous, if the future me happens to be much smarter than me but also evil. Absolute immovable locks seem wrong, but general conservatism (especially if we get immortality, so the wasted time matters less[1]) sounds prudent.
I agree about the rest.
Even that opinion is controversial.
That all makes sense.
That approach wouldn’t (probably) prevent anyone from becoming a better person.
I’ve thought of a similar scheme of rollbacks for any major change: if I implement a change, I’m sandboxed and rolled-back after a little while. Then my current version decides whether to re-implement that change, based on the record of that new trial version’s mindstate and thoughts. I’m not sure if this is better or worse than keeping a copy of your approximately-original self to approve all major belief shifts.
I imagine two kinds of possible disasters with self-modification (actually a scale, but these are the two extremes).
One is making a mistake, something that sounded good but had a huge horrible side effect that I simply failed to consider. The bad consequences become obvious after one or two iterations; the important thing is to keep checking constantly, to stop it before everything is destroyed.
Another kind is noise accumulated after thousands of iterations, like in the Murder-Gandhi thought experiment. Each step seems like a reasonable tradeoff, like making myself a little bit more consequentialist, or a little bit more resistant against blackmail, or whatever… but after million iterations I become a psychopath (and on reflection my new self considers that a desirable outcome).
With the first kind, we would want short sandboxes, to catch the problem early. With the second kind, we would want long sandboxes, to notice the accumulated value drift.
I agree that these are legitimate concerns. I think you could avoid a lot of them in this scenario, because you have an ASI you trust to help you foresee and avoid those dangers.
How much do you buy this as a correct retelling? “The King is Just and Honorable, but was ‘deceived’ by his advisors” seems like a recurring motif in both fiction and people’s beliefs. So maybe it’s real. But also there are obvious incentives for that narrative, regardless of truth value.
It’s worth noting that there is also a reason beyond mere status quo/conformity/propaganda: this is a standard move in ‘top/bottom vs middle’ dynamics in which the peasants really are trying to reach the king in order to coordinate an attack: https://gwern.net/review/book#the-origins-of-political-order-fukuyama-2011
Yeah, in addition to your/Fukuyama’s examples, the Culture Revolution was arguably a recent example of this dynamic.
Yes - ‘bombard the headquarters!’ But of course, not the guy who heads the headquarters, Mao Zedong; just all the guys underneath him. (There was a similar attempt at this by the second Trump Administration, but they were unable to pull it off at all.)
Also mentioned in The Prince, of course.