A couple weeks ago, Vladimir Nesov stirred up the biggest hornet’s nest I’ve ever seen on LW by introducing us to the Counterfactual Mugging scenario.
If you didn’t read it the first time, please do—I don’t plan to attempt to summarize. Further, if you don’t think you would give Omega the $100 in that situation, I’m afraid this article will mean next to nothing to you.
So, those still reading, you would give Omega the $100. You would do so because if someone told you about the problem now, you could do the expected utility calculation 0.5U(-$100)+0.5U(+$10000)>0. Ah, but where did the 0.5s come from in your calculation? Well, Omega told you he flipped a fair coin. Until he did, there existed a 0.5 probability of either outcome. Thus, for you, hearing about the problem, there is a 0.5 probability of your encountering the problem as stated, and a 0.5 probability of your encountering the corresponding situation, in which Omega either hands you $10000 or doesn’t, based on his prediction. This is all very fine and rational.
So, new problem. Let’s leave money out of it, and assume Omega hands you 1000 utilons in one case, and asks for them in the other—exactly equal utility. What if there is an urn, and it contains either a red or a blue marble, and Omega looks, maybe gives you the utility if the marble is red, and asks for it if the marble is blue? What if you have devoted considerable time to determining whether the marble is red or blue, and your subjective probability has fluctuated over the course of you life? What if, unbeknownst to you, a rationalist community has been tracking evidence of the marble’s color (including your own probability estimates), and running a prediction market, and Omega now shows you a plot of the prices over the past few years?
In short, what information do you use to calculate the probability you plug into the EU calculation?
This is probably mean of me, but I’d prefer if the next article about Omega’s various goings-on set out to explain why I should care about what the rational thing to do in Omega-ish situations.
So, those still reading, you would give Omega the $100.
It’s a little too strong, I think you shouldn’t give away the $100, because you are just not reflectively consistent. It’s not you who could’ve ran the expected utility calculation to determine that you should give it away. If you persist, by the time you must do the action it’s not in your interest anymore, it’s a lost cause. And that is a subject of another post that has been lying in draft form for some time.
If you are strong enough to be reflectively consistent, then …
In short, what information do you use to calculate the probability you plug into the EU calculation?
You use your prior for probabilistic valuation, structured to capture expected subsequent evidence on possible branches. According to evidence and possible decisions on each branch, you calculate expected utility of all of the possible branches, find a global feasible maximum, and perform a component decision from it that fits the real branch. The information you have doesn’t directly help in determining the global solution, it only shows which of the possible branches you are on, and thus which role should you play in the global decision, that mostly applies to the counterfactual branches. This works if the prior/utility is something inside you, worse if you have to mine information from the real branch for it in the process. Or, for more generality, you can consider yourself cooperating with your counterfactual counterparts.
The crux of the problem is that you care about counterfactuals; once you attain this, the rest is business as usual. When you are not being reflectively consistent, you let the counterfactual goodness slip away from your fingers, turning to myopically optimizing only what’s real.
Let’s empty out my draft folder then....
Counterfactual Mugging v. Subjective Probability
A couple weeks ago, Vladimir Nesov stirred up the biggest hornet’s nest I’ve ever seen on LW by introducing us to the Counterfactual Mugging scenario.
If you didn’t read it the first time, please do—I don’t plan to attempt to summarize. Further, if you don’t think you would give Omega the $100 in that situation, I’m afraid this article will mean next to nothing to you.
So, those still reading, you would give Omega the $100. You would do so because if someone told you about the problem now, you could do the expected utility calculation 0.5U(-$100)+0.5U(+$10000)>0. Ah, but where did the 0.5s come from in your calculation? Well, Omega told you he flipped a fair coin. Until he did, there existed a 0.5 probability of either outcome. Thus, for you, hearing about the problem, there is a 0.5 probability of your encountering the problem as stated, and a 0.5 probability of your encountering the corresponding situation, in which Omega either hands you $10000 or doesn’t, based on his prediction. This is all very fine and rational.
So, new problem. Let’s leave money out of it, and assume Omega hands you 1000 utilons in one case, and asks for them in the other—exactly equal utility. What if there is an urn, and it contains either a red or a blue marble, and Omega looks, maybe gives you the utility if the marble is red, and asks for it if the marble is blue? What if you have devoted considerable time to determining whether the marble is red or blue, and your subjective probability has fluctuated over the course of you life? What if, unbeknownst to you, a rationalist community has been tracking evidence of the marble’s color (including your own probability estimates), and running a prediction market, and Omega now shows you a plot of the prices over the past few years?
In short, what information do you use to calculate the probability you plug into the EU calculation?
This is probably mean of me, but I’d prefer if the next article about Omega’s various goings-on set out to explain why I should care about what the rational thing to do in Omega-ish situations.
It’s a little too strong, I think you shouldn’t give away the $100, because you are just not reflectively consistent. It’s not you who could’ve ran the expected utility calculation to determine that you should give it away. If you persist, by the time you must do the action it’s not in your interest anymore, it’s a lost cause. And that is a subject of another post that has been lying in draft form for some time.
If you are strong enough to be reflectively consistent, then …
You use your prior for probabilistic valuation, structured to capture expected subsequent evidence on possible branches. According to evidence and possible decisions on each branch, you calculate expected utility of all of the possible branches, find a global feasible maximum, and perform a component decision from it that fits the real branch. The information you have doesn’t directly help in determining the global solution, it only shows which of the possible branches you are on, and thus which role should you play in the global decision, that mostly applies to the counterfactual branches. This works if the prior/utility is something inside you, worse if you have to mine information from the real branch for it in the process. Or, for more generality, you can consider yourself cooperating with your counterfactual counterparts.
The crux of the problem is that you care about counterfactuals; once you attain this, the rest is business as usual. When you are not being reflectively consistent, you let the counterfactual goodness slip away from your fingers, turning to myopically optimizing only what’s real.