Criticism against “unembedded FDT” doesn’t apply to FDT
TL;DR: Some criticisms aimed at FDT are actually aimed at a self-contradictory “unembedded FDT,” and are therefore irrelevant to any refutation of FDT.
Functional Decision Theory (FDT) is a decision theory for a rational agent X who holds the rational belief “the probability that agent Y correctly predicts my actions is very high.” Since X is rational, this belief must be grounded in some causal entanglement[1] — known to X — between X’s actions and Y’s predictions. But since X’s action doesn’t precede Y’s prediction, both must be caused by something else.
In other words: X’s action must be the causal effect of something (call it X’s code), and X’s code must be knowable to Y, and therefore part of Y’s universe, which happens to be the same as X’s. In other words: X is an embedded agent.
So FDT only applies to embedded agents. Why does this matter? Because many thought experiments include an unembedded, “God’s-eye view” premise. Whenever someone says, “Imagine you’re in scenario S, and you know it’s not a simulation,” they’re asking you to imagine a universe that is not itself part of anything bigger. You know there’s “nothing else” out there. But knowing that requires you to be “bigger” than the whole thing — which is incompatible with embeddedness.
1. An unembedded agent imagines a universe that is not a simulation.
2. An unembedded agent imagines a simulated universe.
3. An unembedded agent imagines a universe that is not a simulation, and imagines themselves acting on it (as when playing a video game).
4. An embedded agent who knows they are in a non-simulated universe is a contradiction.
Does this mean a rational embedded agent must remain agnostic, at all times, about whether they’re in a simulation? Not necessarily. The simulations we’re familiar with (dreams, etc.) are usually imperfect enough that we can often tell something is off. And we’ve never heard of anyone who, as in the Hindu story, lived through a fifty-year illusion only to wake up the moment before it began. So when asked, “Do you think you’re in a simulation right now?”, it isn’t absurd to answer, “I’m 99% confident I’m not.”[2]
But the moment we devise a scenario where Omega can predict our actions almost perfectly, the probability that we are inside Omega’s simulation goes up. And you can’t tack on “but you know you are not in a simulation,” or ask “OK, but if you are in the real situation, what do you do?”, because the whole point of FDT is to prescribe a policy to an embedded agent who can never know that they are not in a simulation.
A post from yesterday[3] tries to make FDT palatable to people who, presumably, don’t see themselves as embedded agents. This isn’t a criticism of the audience! Seeing oneself as an embedded agent is genuinely hard — I fail at it all the time, despite actively trying. But the palatable version of FDT doesn’t try to change the audience’s understanding of embeddedness, and so it turns out to be itself incompatible with embeddedness.
Here is an embedded reformulation of the Bomb* scenario from that post, along with an embedded-FDT reply:
Facts:
1. You are a rational, embedded agent with the rational beliefs 2, 3, 4, and (5)
2. A hypothetical scenario S is such that doing A maximizes your utility.
3.Omega can predict your behavior in scenario S with a failure rate of one in a trillion trillion.
4. You are faced with an instance of scenario S in which Omega has predicted you won’t do A.
There is an implicit premise that needs to be added for this to make sense:
5. You know that Omega isn’t changing the rules of the game and/or trying to kill you and/or doing any other non-specified shenanigans.
What do you do?
Embedded FDT’s answer: The scenario rules out everything we would normally assume if a similar something like this happened to us in real life: that our estimate of Omega’s predictive skill was wrong, that Omega is murdering me for fun, or that we’ve misunderstood something.
With all of that off the table, the only remaining explanation is that “I” am in a counterfactual simulation[4] — say, a hallucination Omega has induced in me to see what I’d do. So not only does the usual updateless argument for doing A go through, but I might even conclude that my decision is causally connected to what I’ll observe when Omega runs the real thing.
- ^
Reading “causal entanglement” in an FDT context might come across as strange. Note, however, that I am not talking about how X makes a decision, but about why the statement “Y can predict X’s decisions” is true.
- ^
I’m not claiming this number is a particularly rational answer to the question — only that it isn’t obviously wrong, and that rejecting it would require taking several other things into account.
- ^
A post definitely worth reading! I mention it simply because it prompted me to think about this.
- ^
Stuart Armstrong makes a similar point:
If the predictor is simulating you and asking “would you go left after reading a prediction that you are going right”, then you should go left; because, by the probabilities in the setup, you are almost certainly a simulation.
I mean, unembedded agents are not necessarily unpredictable? You don’t need to fully instantiate an agent to predict it, e.g. 100 parameter N-grams model can predict human choice in Rock Paper Scissors, after observing some number of moves. So, even if agent and the universe are separate and converse through a narrow channel, you need to track logical/correlational dependencies of your choice with stuff in universe.
You don’t need high prediction accuracy for that, any edge would work. Also, it’s more than that. E.g. you can describe EDT like that, and it does not pay in Counterfactual mugging for example.
Thanks a lot for your thoughts!
I am quite confused about this issue, but I’ll keep defending my current view, hoping that you can refute it or give me more clarity. My impression is that “FDT for embedded agents” should be distinguished from “things that look like FDT in other scenarios”. And I think that part of the reason why many find FDT absurd is because they are thinking about FDT outside of the scenario in which it makes sense.
Applied to your first example: If I am playing Rock Paper Scissors against a very simple predictor that can find patterns in the sequence of R/P/S that my brain outputs when I use my System 1… then I think this is not a real predictor of my whole agency, because I am a more complex agent than just my (sub-)System 1. Whenever I want, I can decide to use a different algorithm to output R/P/S, including using external objects to output true randomness. So if we transform this into a Newcombe-like scenario, I think the rational strategy would be to two-box, because being able to predict my System 1 isn’t able to predict me as an agent.
Or in other words: If I see “Omega” predicting the behavior of agents known to it only through narrow channels, I don’t have enough information to gauge how good this “Omega” is at predicting me.
One could insist in thinking about the scenario in which I am limited to outputting R/S/P using my System 1. In this case I am predictable, but I am unsure of whether this is a scenario where FDT can be applied. It seems to me to be more similar to something like the fact that if a human walks in the desert, they’ll end up walking in circles because their steps will be slightly asymmetrical in a systematical way. This can be predicted and exploited, of course, but I’m not sure if this is relevant for Decision Theory, or a quirk of the hardware that a human has to use to output their decision onto the universe.
True. This line is wrong and I’ll change it.
Could you elaborate on what you mean here?
Predictor: He will think I will not be able to predict him, so he will two box.
You: It will not be able to predict me, so I two box.
Harmony
But the “predictor” is not basing their strategy in any deep knowledge about me, right? So their strategy can’t get a 99% success rate as as stipulated in a Newcombe scenario.
Otherwise we should one-box every time any one claims to predict our actions, no matter how good they are at predicting.
(Again, I am confused, so I am very open to being convinced of what you are saying. It’s just that I don’t get it)
Well, it’s a bit tricky, if you think the predictor is bad, and two box, then you become predictable, and the predictor becomes better. There is some logical time game of tag, but with very good predictors this is irrelevant, yeah.
You can also one up bad predictors, by looking like you are going to one box and then two boxing, but that’s distinct from randomizing. There is no good formalism for this, c.f. Schelling points.