Locking in your current values so you can’t change your mind later seems like a very weird move. It doesn’t seem like most people would do that.
I could imagine e.g. religious people doing that, to make sure they won’t apostatize.
Heck, I would probably do something similar myself—not exactly locking my current values, but more like setting up some safeguard against a possibility of doing something that my current self would consider abhorrent. Something like “hey, I don’t want to micromanage my future self, but if the future becomes a universe literally full of paperclips, or torture chambers, then something went horribly wrong, and maybe my mind should be reset to its current state and given a chance to reflect on how that possibly happened”.
It is hard to say what is an acceptable change and what is not. If I was an ancient Roman, I would like to keep myself an option to abolish slavery, no matter how weird it would feel at the start. So I would be like “if for some reason the entire humanity decides to transform themselves into intelligent bunnies, well it seems weird to my current self, my maybe from the perspective of my wiser self it somehow makes perfect sense”. I wouldn’t want to limit my future self at deciding e.g. whether to colonize the galaxies or just build a huge Dyson sphere with a Matrix at the center of our galaxy. I have a preference for letting people freely do whatever they want, but I think it would make sense to limit some harmful options, and I would let my wiser future self to decide on the exact rules. I realize that both making and not making safeguards is a very dangerous option. Would probably err on the side of making the safeguards but making them updatable, like “the future me has to convince the current me to agree on removing the safeguard”, and of course even that is dangerous, if the future me happens to be much smarter than me but also evil. Absolute immovable locks seem wrong, but general conservatism (especially if we get immortality, so the wasted time matters less[1]) sounds prudent.
That approach wouldn’t (probably) prevent anyone from becoming a better person.
I’ve thought of a similar scheme of rollbacks for any major change: if I implement a change, I’m sandboxed and rolled-back after a little while. Then my current version decides whether to re-implement that change, based on the record of that new trial version’s mindstate and thoughts. I’m not sure if this is better or worse than keeping a copy of your approximately-original self to approve all major belief shifts.
I imagine two kinds of possible disasters with self-modification (actually a scale, but these are the two extremes).
One is making a mistake, something that sounded good but had a huge horrible side effect that I simply failed to consider. The bad consequences become obvious after one or two iterations; the important thing is to keep checking constantly, to stop it before everything is destroyed.
Another kind is noise accumulated after thousands of iterations, like in the Murder-Gandhi thought experiment. Each step seems like a reasonable tradeoff, like making myself a little bit more consequentialist, or a little bit more resistant against blackmail, or whatever… but after million iterations I become a psychopath (and on reflection my new self considers that a desirable outcome).
With the first kind, we would want short sandboxes, to catch the problem early. With the second kind, we would want long sandboxes, to notice the accumulated value drift.
I agree that these are legitimate concerns. I think you could avoid a lot of them in this scenario, because you have an ASI you trust to help you foresee and avoid those dangers.
I could imagine e.g. religious people doing that, to make sure they won’t apostatize.
Heck, I would probably do something similar myself—not exactly locking my current values, but more like setting up some safeguard against a possibility of doing something that my current self would consider abhorrent. Something like “hey, I don’t want to micromanage my future self, but if the future becomes a universe literally full of paperclips, or torture chambers, then something went horribly wrong, and maybe my mind should be reset to its current state and given a chance to reflect on how that possibly happened”.
It is hard to say what is an acceptable change and what is not. If I was an ancient Roman, I would like to keep myself an option to abolish slavery, no matter how weird it would feel at the start. So I would be like “if for some reason the entire humanity decides to transform themselves into intelligent bunnies, well it seems weird to my current self, my maybe from the perspective of my wiser self it somehow makes perfect sense”. I wouldn’t want to limit my future self at deciding e.g. whether to colonize the galaxies or just build a huge Dyson sphere with a Matrix at the center of our galaxy. I have a preference for letting people freely do whatever they want, but I think it would make sense to limit some harmful options, and I would let my wiser future self to decide on the exact rules. I realize that both making and not making safeguards is a very dangerous option. Would probably err on the side of making the safeguards but making them updatable, like “the future me has to convince the current me to agree on removing the safeguard”, and of course even that is dangerous, if the future me happens to be much smarter than me but also evil. Absolute immovable locks seem wrong, but general conservatism (especially if we get immortality, so the wasted time matters less[1]) sounds prudent.
I agree about the rest.
Even that opinion is controversial.
That all makes sense.
That approach wouldn’t (probably) prevent anyone from becoming a better person.
I’ve thought of a similar scheme of rollbacks for any major change: if I implement a change, I’m sandboxed and rolled-back after a little while. Then my current version decides whether to re-implement that change, based on the record of that new trial version’s mindstate and thoughts. I’m not sure if this is better or worse than keeping a copy of your approximately-original self to approve all major belief shifts.
I imagine two kinds of possible disasters with self-modification (actually a scale, but these are the two extremes).
One is making a mistake, something that sounded good but had a huge horrible side effect that I simply failed to consider. The bad consequences become obvious after one or two iterations; the important thing is to keep checking constantly, to stop it before everything is destroyed.
Another kind is noise accumulated after thousands of iterations, like in the Murder-Gandhi thought experiment. Each step seems like a reasonable tradeoff, like making myself a little bit more consequentialist, or a little bit more resistant against blackmail, or whatever… but after million iterations I become a psychopath (and on reflection my new self considers that a desirable outcome).
With the first kind, we would want short sandboxes, to catch the problem early. With the second kind, we would want long sandboxes, to notice the accumulated value drift.
I agree that these are legitimate concerns. I think you could avoid a lot of them in this scenario, because you have an ASI you trust to help you foresee and avoid those dangers.