Firstly, let’s assume that we deny physical continuity of personal identity, but embrace psychological continuity instead. In that case, you don’t want to press the button as it simulation has just as much claim to be the continuation of “you” as the original you, even if each instance knows whether it is in the real world or in a simulation.
The rest of this analysis will assume continuity of physical identity instead. From this perspective, pressing the button reduces your expectation of future utility, but it only does this by changing your epistemic state, rather than actually making things worse for the person who pushed the button. As an example, if I was to completely erase the strongest evidence of global warming from everyone’s mind, I would reduce the expectation of a catastrophe, but not the likelihood of it occurring. Here, we’ve done the same, just by creating simulations so that you no longer are certain whether you are the original or a simulation. Since this is only a change of your epistemic state, you should actually push the button.
This is actually quite an important problem for building an AI as we don’t want it to mess with its expectation.
Firstly, let’s assume that we deny physical continuity of personal identity, but embrace psychological continuity instead. In that case, you don’t want to press the button as it simulation has just as much claim to be the continuation of “you” as the original you, even if each instance knows whether it is in the real world or in a simulation.
The rest of this analysis will assume continuity of physical identity instead. From this perspective, pressing the button reduces your expectation of future utility, but it only does this by changing your epistemic state, rather than actually making things worse for the person who pushed the button. As an example, if I was to completely erase the strongest evidence of global warming from everyone’s mind, I would reduce the expectation of a catastrophe, but not the likelihood of it occurring. Here, we’ve done the same, just by creating simulations so that you no longer are certain whether you are the original or a simulation. Since this is only a change of your epistemic state, you should actually push the button.
This is actually quite an important problem for building an AI as we don’t want it to mess with its expectation.