unembedded agents are not necessarily unpredictable?
I am quite confused about this issue, but I’ll keep defending my current view, hoping that you can refute it or give me more clarity. My impression is that “FDT for embedded agents” should be distinguished from “things that look like FDT in other scenarios”. And I think that part of the reason why many find FDT absurd is because they are thinking about FDT outside of the scenario in which it makes sense.
Applied to your first example: If I am playing Rock Paper Scissors against a very simple predictor that can find patterns in the sequence of R/P/S that my brain outputs when I use my System 1… then I think this is not a real predictor of my whole agency, because I am a more complex agent than just my (sub-)System 1. Whenever I want, I can decide to use a different algorithm to output R/P/S, including using external objects to output true randomness. So if we transform this into a Newcombe-like scenario, I think the rational strategy would be to two-box, because being able to predict my System 1 isn’t able to predict me as an agent.
Or in other words: If I see “Omega” predicting the behavior of agents known to it only through narrow channels, I don’t have enough information to gauge how good this “Omega” is at predicting me.
One could insist in thinking about the scenario in which I am limited to outputting R/S/P using my System 1. In this case I am predictable, but I am unsure of whether this is a scenario where FDT can be applied. It seems to me to be more similar to something like the fact that if a human walks in the desert, they’ll end up walking in circles because their steps will be slightly asymmetrical in a systematical way. This can be predicted and exploited, of course, but I’m not sure if this is relevant for Decision Theory, or a quirk of the hardware that a human has to use to output their decision onto the universe.
You don’t need high prediction accuracy for that, any edge would work.
True. This line is wrong and I’ll change it.
Also, it’s more than that. E.g. you can describe EDT like that, and it does not pay in Counterfactual mugging for example.
So if we transform this into a Newcombe-like scenario, I think the rational strategy would be to two-box, because being able to predict my System 1 isn’t able to predict me as an agent.
Predictor: He will think I will not be able to predict him, so he will two box.
You: It will not be able to predict me, so I two box.
But the “predictor” is not basing their strategy in any deep knowledge about me, right? So their strategy can’t get a 99% success rate as as stipulated in a Newcombe scenario.
Otherwise we should one-box every time any one claims to predict our actions, no matter how good they are at predicting.
(Again, I am confused, so I am very open to being convinced of what you are saying. It’s just that I don’t get it)
Well, it’s a bit tricky, if you think the predictor is bad, and two box, then you become predictable, and the predictor becomes better. There is some logical time game of tag, but with very good predictors this is irrelevant, yeah.
You can also one up bad predictors, by looking like you are going to one box and then two boxing, but that’s distinct from randomizing. There is no good formalism for this, c.f. Schelling points.
Thanks a lot for your thoughts!
I am quite confused about this issue, but I’ll keep defending my current view, hoping that you can refute it or give me more clarity. My impression is that “FDT for embedded agents” should be distinguished from “things that look like FDT in other scenarios”. And I think that part of the reason why many find FDT absurd is because they are thinking about FDT outside of the scenario in which it makes sense.
Applied to your first example: If I am playing Rock Paper Scissors against a very simple predictor that can find patterns in the sequence of R/P/S that my brain outputs when I use my System 1… then I think this is not a real predictor of my whole agency, because I am a more complex agent than just my (sub-)System 1. Whenever I want, I can decide to use a different algorithm to output R/P/S, including using external objects to output true randomness. So if we transform this into a Newcombe-like scenario, I think the rational strategy would be to two-box, because being able to predict my System 1 isn’t able to predict me as an agent.
Or in other words: If I see “Omega” predicting the behavior of agents known to it only through narrow channels, I don’t have enough information to gauge how good this “Omega” is at predicting me.
One could insist in thinking about the scenario in which I am limited to outputting R/S/P using my System 1. In this case I am predictable, but I am unsure of whether this is a scenario where FDT can be applied. It seems to me to be more similar to something like the fact that if a human walks in the desert, they’ll end up walking in circles because their steps will be slightly asymmetrical in a systematical way. This can be predicted and exploited, of course, but I’m not sure if this is relevant for Decision Theory, or a quirk of the hardware that a human has to use to output their decision onto the universe.
True. This line is wrong and I’ll change it.
Could you elaborate on what you mean here?
Predictor: He will think I will not be able to predict him, so he will two box.
You: It will not be able to predict me, so I two box.
Harmony
But the “predictor” is not basing their strategy in any deep knowledge about me, right? So their strategy can’t get a 99% success rate as as stipulated in a Newcombe scenario.
Otherwise we should one-box every time any one claims to predict our actions, no matter how good they are at predicting.
(Again, I am confused, so I am very open to being convinced of what you are saying. It’s just that I don’t get it)
Well, it’s a bit tricky, if you think the predictor is bad, and two box, then you become predictable, and the predictor becomes better. There is some logical time game of tag, but with very good predictors this is irrelevant, yeah.
You can also one up bad predictors, by looking like you are going to one box and then two boxing, but that’s distinct from randomizing. There is no good formalism for this, c.f. Schelling points.