That approach wouldn’t (probably) prevent anyone from becoming a better person.
I’ve thought of a similar scheme of rollbacks for any major change: if I implement a change, I’m sandboxed and rolled-back after a little while. Then my current version decides whether to re-implement that change, based on the record of that new trial version’s mindstate and thoughts. I’m not sure if this is better or worse than keeping a copy of your approximately-original self to approve all major belief shifts.
I imagine two kinds of possible disasters with self-modification (actually a scale, but these are the two extremes).
One is making a mistake, something that sounded good but had a huge horrible side effect that I simply failed to consider. The bad consequences become obvious after one or two iterations; the important thing is to keep checking constantly, to stop it before everything is destroyed.
Another kind is noise accumulated after thousands of iterations, like in the Murder-Gandhi thought experiment. Each step seems like a reasonable tradeoff, like making myself a little bit more consequentialist, or a little bit more resistant against blackmail, or whatever… but after million iterations I become a psychopath (and on reflection my new self considers that a desirable outcome).
With the first kind, we would want short sandboxes, to catch the problem early. With the second kind, we would want long sandboxes, to notice the accumulated value drift.
I agree that these are legitimate concerns. I think you could avoid a lot of them in this scenario, because you have an ASI you trust to help you foresee and avoid those dangers.
That all makes sense.
That approach wouldn’t (probably) prevent anyone from becoming a better person.
I’ve thought of a similar scheme of rollbacks for any major change: if I implement a change, I’m sandboxed and rolled-back after a little while. Then my current version decides whether to re-implement that change, based on the record of that new trial version’s mindstate and thoughts. I’m not sure if this is better or worse than keeping a copy of your approximately-original self to approve all major belief shifts.
I imagine two kinds of possible disasters with self-modification (actually a scale, but these are the two extremes).
One is making a mistake, something that sounded good but had a huge horrible side effect that I simply failed to consider. The bad consequences become obvious after one or two iterations; the important thing is to keep checking constantly, to stop it before everything is destroyed.
Another kind is noise accumulated after thousands of iterations, like in the Murder-Gandhi thought experiment. Each step seems like a reasonable tradeoff, like making myself a little bit more consequentialist, or a little bit more resistant against blackmail, or whatever… but after million iterations I become a psychopath (and on reflection my new self considers that a desirable outcome).
With the first kind, we would want short sandboxes, to catch the problem early. With the second kind, we would want long sandboxes, to notice the accumulated value drift.
I agree that these are legitimate concerns. I think you could avoid a lot of them in this scenario, because you have an ASI you trust to help you foresee and avoid those dangers.