I’m having trouble understanding this. You’re saying that Omega flipped the coin before R started coding, but R doesn’t know that, or the result of the coin flip, right? Then his P(a counterfactual mugging is ongoing) is very low, and P(heads | a counterfactual mugging is ongoing) = P(tails | a counterfactual mugging is ongoing) = 1⁄2. Right?
In that case, his expected utility at the time of coding is maximized by S outputting “give $100” upon encountering Omega. It seems entirely straightforward, and I don’t see what the problem is...
. . . do what my creator would want me to do. In other words, upon receiving input X, S computes the following: suppose S’s creator had enough time and computing power to create a giant lookup table that contains an optimal output for every input S might encounter, what would the entry for X be? Return that as the output.
I don’t know how to define what R “would want” or would think was “optimal”.
What lookup table would R create? If R is a causal decision theorist, R might think: “If I were being counterfactually mugged and Omega’s coin had come up heads, Omega would have already made its prediction about whether S would output ‘give $100’ on the input ‘tails’. So, if I program S with the rule ‘give $100 if tails’, that won’t cause Omega to give me $10000. And if the coin came up tails, that rule would lose me $100. So I will program S with the rule ‘give $0 if tails’.”
R’s expected utility at the time of coding may be maximized by the rule “give $100 if tails”, but R makes decisions by the conditional expected utilities given each of Omega’s possible past predictions, weighted by R’s prior beliefs about those predictions. R’s conditional expected utilities are both maximized by the decision to program S to output “give $0″.
[I deleted my earlier reply, because I was still confused about your questions.]
If, according to R’s decision theory, the most preferred choice involves programming S to output “give $0”, then that is what S would do.
It might be easier to think of the ideal S as consisting of a giant lookup table created by R itself given infinite time and computing power. An actual S would try to approximate this ideal to the best of its abilities.
How should S decide, from its inputs, which R is the creator with the expected utility S’s outputs should be optimal for? Is it the R in the world where Omega’s coin came up heads, or the R in the world where Omega’s coin came up tails?
R would encode its own decision theory, prior, utility function, and memory at the time of coding into S, and have S optimize for that R.
Sorry. I wasn’t trying to ask my questions as questions about how R would make decisions. I was asking questions to try to answer your question about the relationship between exceptionless and timeless decision-making, by pointing out dimensions of a map of ways for R to make decisions. For some of those ways, S would be “timeful” around R’s beliefs or time of coding, and for some of those ways S would be less timeful.
I have an intuition that there is a version of reflective consistency which requires R to code S so that, if R was created by another agent Q, S would make decisions using Q’s beliefs even if Q’s beliefs were different from R’s beliefs (or at least the beliefs that a Bayesian updater would have had in R’s position), and even when S or R had uncertainty about which agent Q was. But I don’t know how to formulate that intuition to something that could be proven true or false. (But ultimately, S has to be a creator of its own successor states, and S should use the same theory to describe its relation to its past selves as to describe its relation to R or Q. S’s decisions should be invariant to the labeling or unlabeling of its past selves as “creators”. These sequential creations are all part of the same computational process.)
I see, you meant that Omega made its prediction before R started coding. In that case, S won’t output “give $100”.
It might be easier to think of the ideal S as consisting of a giant lookup table created by R itself given infinite time and computing power. An actual S would try to approximate this GLT to the best of its abilities. Does that answer your questions?
I’m having trouble understanding this. You’re saying that Omega flipped the coin before R started coding, but R doesn’t know that, or the result of the coin flip, right? Then his P(a counterfactual mugging is ongoing) is very low, and P(heads | a counterfactual mugging is ongoing) = P(tails | a counterfactual mugging is ongoing) = 1⁄2. Right?
In that case, his expected utility at the time of coding is maximized by S outputting “give $100” upon encountering Omega. It seems entirely straightforward, and I don’t see what the problem is...
I don’t know how to define what R “would want” or would think was “optimal”.
What lookup table would R create? If R is a causal decision theorist, R might think: “If I were being counterfactually mugged and Omega’s coin had come up heads, Omega would have already made its prediction about whether S would output ‘give $100’ on the input ‘tails’. So, if I program S with the rule ‘give $100 if tails’, that won’t cause Omega to give me $10000. And if the coin came up tails, that rule would lose me $100. So I will program S with the rule ‘give $0 if tails’.”
R’s expected utility at the time of coding may be maximized by the rule “give $100 if tails”, but R makes decisions by the conditional expected utilities given each of Omega’s possible past predictions, weighted by R’s prior beliefs about those predictions. R’s conditional expected utilities are both maximized by the decision to program S to output “give $0″.
[I deleted my earlier reply, because I was still confused about your questions.]
If, according to R’s decision theory, the most preferred choice involves programming S to output “give $0”, then that is what S would do.
It might be easier to think of the ideal S as consisting of a giant lookup table created by R itself given infinite time and computing power. An actual S would try to approximate this ideal to the best of its abilities.
R would encode its own decision theory, prior, utility function, and memory at the time of coding into S, and have S optimize for that R.
Sorry. I wasn’t trying to ask my questions as questions about how R would make decisions. I was asking questions to try to answer your question about the relationship between exceptionless and timeless decision-making, by pointing out dimensions of a map of ways for R to make decisions. For some of those ways, S would be “timeful” around R’s beliefs or time of coding, and for some of those ways S would be less timeful.
I have an intuition that there is a version of reflective consistency which requires R to code S so that, if R was created by another agent Q, S would make decisions using Q’s beliefs even if Q’s beliefs were different from R’s beliefs (or at least the beliefs that a Bayesian updater would have had in R’s position), and even when S or R had uncertainty about which agent Q was. But I don’t know how to formulate that intuition to something that could be proven true or false. (But ultimately, S has to be a creator of its own successor states, and S should use the same theory to describe its relation to its past selves as to describe its relation to R or Q. S’s decisions should be invariant to the labeling or unlabeling of its past selves as “creators”. These sequential creations are all part of the same computational process.)
I see, you meant that Omega made its prediction before R started coding. In that case, S won’t output “give $100”.
It might be easier to think of the ideal S as consisting of a giant lookup table created by R itself given infinite time and computing power. An actual S would try to approximate this GLT to the best of its abilities. Does that answer your questions?