Hmm, yeah, I think I was wrong. You probably shouldn’t pay for “the second prime is 4,” unless Omega is doing some really dramatic brain-editing in its counterfactual simulation. This highlights where the center of the problem is—the problem isn’t designing the agent, the problem is understanding, in detail, an Omega that evaluates logical counterfactuals.
But, and this is what tripped me up, Omega’s beliefs are also important. Consider the case where Omega is running a search process for complicated false statements. Now you definitely shouldn’t pay Omega, even if you’re a priori uncertain about the statements Omega mugs you with.
This seems to get into problems with what the decision problem actually is. If you’re “updateless” but you already know you’re in this particular counterfactual mugging, you may respond much differently than if you’re merely considering this as one of many possible muggings. Specifically, in the first case, any Omega is a cheapskate Omega if it chooses something like “P and not P”. In the latter case, however, we might know that Omega arrives at absurdities such as this through a fair process which is equally likely to select true sentences. In that case, despite not being updateless enough to be ignorant about “P and not P”, we might go along with the mogging as part of a policy which achieves high payout on average.
Hmm, yeah, I think I was wrong. You probably shouldn’t pay for “the second prime is 4,” unless Omega is doing some really dramatic brain-editing in its counterfactual simulation. This highlights where the center of the problem is—the problem isn’t designing the agent, the problem is understanding, in detail, an Omega that evaluates logical counterfactuals.
But, and this is what tripped me up, Omega’s beliefs are also important. Consider the case where Omega is running a search process for complicated false statements. Now you definitely shouldn’t pay Omega, even if you’re a priori uncertain about the statements Omega mugs you with.
This seems to get into problems with what the decision problem actually is. If you’re “updateless” but you already know you’re in this particular counterfactual mugging, you may respond much differently than if you’re merely considering this as one of many possible muggings. Specifically, in the first case, any Omega is a cheapskate Omega if it chooses something like “P and not P”. In the latter case, however, we might know that Omega arrives at absurdities such as this through a fair process which is equally likely to select true sentences. In that case, despite not being updateless enough to be ignorant about “P and not P”, we might go along with the mogging as part of a policy which achieves high payout on average.