Yeah, it matches wat @mesaoptimizer said, I believe. I was reluctant to post my view, but thought it could be helpful anyway :)
Great question! I’m wondering the same thing now. I, for one, had to be reasoned into it. It does feel like it “clicked”, so to speak, but I doubt whether anyone has this intuition naturally.
I would be willing to discuss FDT more, if you’d like (in a separate post, of course).
Regarding (2), I interpret Soares’ point as follows: there’s a “CDT intuition” and an “FDT intuition” of how to evaluate counterfactuals. Let’s just take the Bomb problem as an example.
The CDT intuition deals with which action has the best causal consequences at one given point in time. In Bomb, you can either Left-box or Right-box. There’s a bomb in Left, so Left-boxing causes you to die painfully. Right-boxing costs only $100, so right-boxing wins.
The FDT intuition deals with which decision is the best outcome of your decision procedure. Your decision procedure is a function, and could be implemented more than once. In Bomb, it’s implemented both in your head and in the predictor’s head (she executed it to predict what you would do). Your decision to Left-box or to Right-box therefore happens twice—and, since your decision procedure is a function, it’s necessarily the same on both events—and you have to look at the causal consequences of both events. Left-boxing causes the predictor to not put a bomb in Left and you to not lose $100, while Right-boxing causes the predictor to put a bomb in Left and you to lose $100. Left-boxing wins.
P.S. A natural thing to respond here is something like: “But you already see a Bomb in Left, so the FDT intuition makes no sense!” But note that, since the predictor simulates you in order to make her prediction, you don’t actually know whether you are the “real you” or the “simulated you”, since the simulated you observes the exact same (relevant) things as the real you. (If not, then the simulation would not be accurate and there would be no subjunctive dependence.) So in this intuition, observing the bomb does not actually mean there is a bomb, since you could be in a simulation. In fact, you are, at different points in time, both in the simulation and in the “real” situation, and you have to make a decision that happens in and makes the best of both these situations.
“In your problem description you said you receive the letter”
True, but the problem description also specifies subjunctive dependence between the agent and the predictor. When the predictor made her prediction the letter isn’t yet sent. So the agent’s decision influences whether or not she gets the letter.
“This intuition is actually false for perfect predictors.”
I agree (and have written extensively on the subject). But it’s the prediction the agent influences, not the presence of the termite infestation.
Given that you receive the letter, paying is indeed evidence for not having termites and winning $999,000. EDT is elegant, but still can’t be correct in my view. I wish it were, and have attempted to “fix” it.
My take is this. Either you have the termite infestation, or you don’t.
Say you do. Then
being a “payer” means you never receive the letter, as both conditions are false. As you don’t receive the letter, you don’t actually pay, and lose the $1,000,000 in damages.
being a “non-payer” means you get the letter, and you don’t pay. You lose $1,000,000.
Say you don’t. Then
payer: you get the letter, pay $1,000. You lose $1,000.
non-payer: you don’t get the letter, and don’t pay $1,000. You lose nothing.
Being a payer has the same result when you do have the termites, but is worse when you don’t. So overall, it’s worse. Being a payer or a non-payer only influences whether or not you get the letter, and this view is more coherent with the intuition that you can’t possibly influence whether or not you have a termite infestation.
XOR Blackmail is (in my view) perhaps the clearest counterexample to EDT:
An agent has been alerted to a rumor that her house has a terrible termite infestation that would cost her $1,000,000 in damages. She doesn’t know whether this rumor is true. A greedy predictor with a strong reputation for honesty learns whether or not it’s true, and drafts a letter:
I know whether or not you have termites, and I have sent you this letter iff exactly one of the following is true: (i) the rumor is false, and you are going to pay me $1,000 upon receiving this letter; or (ii) the rumor is true, and you will not pay me upon receiving this letter.
The predictor then predicts what the agent would do upon receiving the letter, and sends the agent the letter iff exactly one of (i) or (ii) is true. Thus, the claim made by the letter is true. Assume the agent receives the letter. Should she pay up?
(Styling mine, not original.) EDT pays the $1,000 for nothing: it has absolutely no influence on whether or not the agent’s house is infested with termites.
Alright, thanks for your answer!
Is it necessary to be able to work during all MIRI office hours, or is it enough if my hours are partially compatible? My time difference with MIRI is 9 hours, but I could work in the evening (my time) every now and then.
Btw, thanks for your comment! I edited my post with respect to fair problems.
I think this is potentially an overly strong criteria for decision theories—we should probably restrict to something like the problems to a fair problem class, else we end up with no decision theory receiving any credence.
Good point, I should have mentioned that in my article. (Note that XOR Blackmail is definitely a fair problem (not that you are claiming otherwise)).
I also think “wrong answer” is doing a lot of work here.
I at least in part agree here. This is why I picked XOR Blackmail, because it has such an obvious right answer. That’s an intuition, but that’s also true for some of the points made in favor of The Evidentialist’s Wager to begin with.
Thanks for the comment!
There is a false dichotomy in the argument basing the conclusion only on the options CDT or EDT, when in fact both are wrong.
I wouldn’t say there’s a false dichotomy: the argument works fine if you also have credence in e.g. FDT. It just says that altruistic, morally motivated agents should favor EDT over CDT. (However, as I have attempted to demonstrate, 2 premises of the argument don’t hold up.)
Suppose you know instead that Omega was miserly and almost all of the people who one-box don’t get offered the opportunity to play—let’s say every two-boxer gets to play but only 0.01% of one-boxers. Should you still choose to 1-box if presented with the opportunity of playing?
Interesting. No, because a 0.01% probability of winning $1,000,000 gives me an expected $100, whereas two-boxing gives me $1,000.
Your original description doesn’t specify subjunctive dependence, which is a critical component of the problem.
Heighn’s response to this argument is that this is a perfectly fine prescription.
Note that omnizoid hasn’t checked with me whether this is my response, and if he had, I would have asked him to specify the problem more. In my response article, I attempt to specify the problem more, and with that particular specification, I do indeed endorse FDT’s decision.
I’m surprised Wei Dai thinks this is a fair point. I disagree entirely with it: FDT is a decision theory and doesn’t in and of itself value anything. The values need to be given by a utility function.
Consider the Psychological Twin Prisoner’s Dilemma. Given the utility function used there, the agent doesn’t value the twin at all: the agent just wants to go home free as soon as possible. FDT doesn’t change this: it just recognizes that the twin makes the same decision the agent does, which has bearing on the prison time the agent gets.
...which makes the Procreation case an unfair problem. It punishes FDT’ers specifically for following FDT. If we’re going to punish decision theories for their identity, no decision theory is safe. It’s pretty wild to me that @WolfgangSchwarz either didn’t notice this or doesn’t think it’s a problem.
A more fair version of Procreation would be what I have called Procreation*, where your father follows the same decision theory as you (be it FDT, CDT or whatever).
Oh wait, of course, in this problem, Omega doesn’t simulate the agent. Interesting! I’ll have to think about this more :-)
In what ways is FDT broken?
I also wonder whether a different problem was intended.
Thanks for the link!
And hmm, it seems to me FDT one-boxes on ASP, as that gives the most utility from the subjunctive dependence perspective.
Why would Omega put $0 in the second box? The problem statement specifies Omega puts $100 in both boxes if she predicts you will two-box!
Why do you believe this is a scenario in which you are definitely not in a simulation?