Answer: It appears to be sufficient that the agent only knows that Omega is always correct. If this is the case, then a simulating-Omega and some-other-method-Omega are indistinguishable, so the agent can freely assume simulation.
This is false. If there is not a conscious simulation running, the agent will know he is not a simulation, and will two box.
Objection 2. The argument does not work for the problems where Omega is not always correct, but correct with, say, 90% probability.
As long as the probability is sufficiently high, and the agent is sufficiently uncertain of whether or not he is the simulation, it works fine.
Answer: The agent in the simulation has exactly the same experiences as the agent outside, so it is the same self, so it values the Omega-offered utilons the same.
If the agent is selfish, and his sense of identity is such that he doesn’t consider the being he is a simulation of himself, the simulated self will not care about the non-simulated self.
I admit it does seem a bit weird to have a utility function that depends on something you have no way of knowing. It’s not impossible, though.
This is false. If there is not a conscious simulation running, the agent will know he is not a simulation, and will two box.
As long as the probability is sufficiently high, and the agent is sufficiently uncertain of whether or not he is the simulation, it works fine.
If the agent is selfish, and his sense of identity is such that he doesn’t consider the being he is a simulation of himself, the simulated self will not care about the non-simulated self.
I admit it does seem a bit weird to have a utility function that depends on something you have no way of knowing. It’s not impossible, though.