If the AI is allowed to pass information to the resetted version, s/he will have to spend more and more time to assimilate that information. In the end, its utility will decrease to 0, unless he is given more time to assimilate the information. And the more time you give it, the more it’s probable that s/he is going to be willing not to be resetted.
Also, moral “quantum event” could always happen: that is, the willingness not to be resetted could always emerge even if the initial probability was very very low.
In symbols, if S(t) is the state of the world at time t, t0 is the time of the upload, U = S(t0), and and P() the probability that the AI accepts a reset:
for the AI to have a utility, it’s needed that:
U =/= S(t > t0)
but, even if P(U) << 1, we cannot enforce that:
P(U) = P(S(t))
for any t > t0, since information can be encoded in the state of the world.
If the AI is allowed to pass information to the resetted version, s/he will have to spend more and more time to assimilate that information. In the end, its utility will decrease to 0, unless he is given more time to assimilate the information. And the more time you give it, the more it’s probable that s/he is going to be willing not to be resetted. Also, moral “quantum event” could always happen: that is, the willingness not to be resetted could always emerge even if the initial probability was very very low. In symbols, if S(t) is the state of the world at time t, t0 is the time of the upload, U = S(t0), and and P() the probability that the AI accepts a reset:
for the AI to have a utility, it’s needed that:
U =/= S(t > t0)
but, even if P(U) << 1, we cannot enforce that:
P(U) = P(S(t))
for any t > t0, since information can be encoded in the state of the world.