The reasoning is different because the problem is different.
The simulated agent and yourself were not subjected to the same problem. Therefore you can perfectly precommit to different decisions. TDT does not automatically take the same decisions to problems that merely kinda look the same. They have to actually be the same. There may be specific reasons why TDT would make the same decision, but I doubt it.
Now on to the examples:
###Newcomb’s problem
Omega ran a simulation of Newcomb’s problem, complete with a TDT agent in it. The simulated TDT agent obviously one boxed, and got the million. If you run TDT yourself, you also know it. Now, Omega tells you of this simulation, and tells you to chose your boxes. This is not Newcomb’s problem. If it was, deciding to 2 box would cause box B to be empty!
CDT would crudely assume that 2 boxing gets it $1000 more than 1 boxing. TDT on the other hand knows the simulated box B (and therefore the real one as well) has the million, regardless of its current decision.
###10 boxes problem
Again, the simulated problem and the real one aren’t the same. If there were, choosing box 1 with probability 1 would cause box 2 to have the million. Because it’s not the same problem, even TDT should be allowed to precommit different decision. The point of TDT is to foresee the consequences of its precommitments. It will therefore know that its precommitment in the real problem doesn’t have any influence to its precommitment (and therefore the outcome) in the simulated one. This lack of influence allows it to fall back on CDT reasoning.
The simulated problem and the actual problem don’t have to actually be the same—just indistinguishable from the point of view of the agent.
Omega avoids infinite regress because the actual contents of the boxes are irrelevant for the purposes of the simulation, so no sub-simulation is necessary.
Okay. So, what specific mistake TDT does that would prevent it to distinguish the two problems? What does it lead it to think “If I precommit X in problem 1, I have to precommit X in problem 2 as well”.
(If the problems aren’t the same, of course Omega can avoid infinite regress. And if there is unbounded regress, we may be able to find a non-infinite solution by looping the regress over itself. But then the problems (simulated an real) are definitely the same.)
In the simulated problem the simulated agent is presented with the choice but never gets the reward; for all it matters both boxes can be empty. This means that Omega doesn’t have to do another simulation to work out what’s in the simulated boxes.
The infinite regress is resolvable anyway—since each TDT agent is facing the exact same problem, their decisions must be identical, hence TDT one-boxes and Omega knows this.
The infinite regress is resolvable anyway—since each TDT agent is facing the exact same problem, their decisions must be identical, hence TDT one-boxes and Omega knows this.
Of course.
Now there’s still the question of the perceived difference between the simulated problem and the real one (I assume here that you should 1 box in the simulation, and 2 box in the real problem). There is a difference, how come TDT does not see it? A Rational Decision Theory would —we humans do. Or if it can see it, how come can’t it act on it? RDT could. Do you concede that TDT does and can, or do you still have doubts?
Due to how the problem is set up, you can’t notice the difference until after you’ve made your decision. The only reason other decision theories know they’re not in the simulation is because the problem explicitly states that a TDT agent is simulated, which means it can’t be them.
The only reason other decision theories know they’re not in the simulation is because the problem explicitly states that a TDT agent is simulated, which means it can’t be them.
That’s false. Here is a modified version of the problem:
Omega presents the usual two boxes A and B and announces the following. “Before you entered the room, I ran a simulation of Newcomb’s problem as presented to you. If your simulated twin 2-boxed then I put nothing in Box B. If your simulated twin 1-boxed, I put $1 million in Box B. In any case, I put $1000 in Box A. Now please 1-box or 2-box.”
Even if you’re not running TDT, the simulated agent is running the same decision algorithm as you are. If that was the reason why TDT couldn’t tell the difference, well, now no one can. However you and I can make the difference. The simulated problem is obviously different:
Omega presents the usual two boxes A and B and announces the following. “I am subjecting you to Newcomb’s problem. Now please 1-box or 2-box”.
Really, the subjective difference between the two problems should be obvious to any remotely rational agent.
(Please let me know if you agree up until that point. Below, I assume you do.)
I’m pretty sure the correct answers for the two problems (my modified version as well as the original one) are 1-box in the simulation, 2-box in the real problem. (Do you still agree?)
So. We both agree that RDT (Rational Decision Theory) 1-boxes in the simulation, and 2-boxes in the real problem. CDT would 2-box in both, and TDT would 1-box in the simulation while in the real problem it would…
2-box? I think so.
1-box? Supposedly because it can’t tell simulation from reality. Or rather, it can’t tell the difference between Newcomb’s problem and the actual problem. Even though RDT does. (riiight?) So again, I must ask, why not? I need a more specific answer than “due to how the problem is set up”. I need you to tell me what specific kind of irrationality TDT is committing here. I need to know its specific blind spot.
Oh. So this is indeed a case of “If you’re running TDT, I screw you, otherwise you walk free”. The way I understand this, TDT one boxes because it should.
If TDT cannot perceive the difference between the original problem and the simulation, this is because it is actually the same problem. For all it knows, it could be in the simulation (the simulation argument would say it is). There is an infinite regress, solved by the fact that all agents in all simulation level will have taken the same decision, because they ran the same decision algorithm. If they all 2-box, they get $1000, while if they all 1-box, they get the million (and no more).
Now, if you replace “an agent running TDT” by “you” (like, a fork of you started from a snapshot taken 3 seconds ago), the correct answer is always to 1-box, because then the problem is equivalent to the actual Newcomb’s problem.
Well, in the problem you present here TDT would 2-box, but you’ve avoided the hard part of the problem from the OP, in which there is no way to tell whether you’re in the simulation or not (or at least there is no way for the simulated you to tell), unless you’re running some algorithm other than TDT.
To get back to the exact original problem as stated by the OP, I only need to replace “you” by “an agent running TDT”, and “your simulated twin” by “the simulated agent”. Do we agree?
Assuming we do agree, are you telling me the hard part is in that change? Are you telling me that TDT would 1-box in the original problem, even though it 2-boxes on my problem?
WHYYYYY?
in which there is no way to tell whether you’re in the simulation or not
Wait a minute, what exactly do you mean by “you”? TDT? or “any agent whatsoever”? If it’s TDT alone why? If I read you correctly, you already agree that’s it’s not because Omega said “running TDT” instead of “running WTF-DT”. If it’s “any agent whatsoever”, then are you really sure the simulated and real problem aren’t actually the same? (I’m sure they aren’t, but, just checking.)
Wait a minute, what exactly do you mean by “you”? TDT? or “any agent whatsoever”? If it’s TDT alone why? If I read you correctly, you already agree that’s it’s not because Omega said “running TDT” instead of “running WTF-DT”. If it’s “any agent whatsoever”, then are you really sure the simulated and real problem aren’t actually the same? (I’m sure they aren’t, but, just checking.)
Well, no, this would be my disagreement: it’s precisely because Omega told you that the simulated agent is running TDT that only TDT could or could not be the simulation; the simulated and real problem are, for all intents and purposes, identical (Omega doesn’t actually need to put a reward in the simulated boxes, because he doesn’t need to reward the simulated agent, but both problems appear exactly the same to the simulated and real TDT agents).
I’ll need to write a full discussion post about that at some point. There is one crucial difference besides “I’m TDT” and “I’m CDT”. It’s “The simulated agent uses the same decision theory” and “The simulated agent does not use the same decision theory”.
That’s not exactly the same problem, and I think that is the whole point.
This is CDT reasoning, AKA causal reasoning. Or in other words, how do you not use the same reasoning in the original Newcombe problem?
The reasoning is different because the problem is different.
The simulated agent and yourself were not subjected to the same problem. Therefore you can perfectly precommit to different decisions. TDT does not automatically take the same decisions to problems that merely kinda look the same. They have to actually be the same. There may be specific reasons why TDT would make the same decision, but I doubt it.
Now on to the examples:
###Newcomb’s problem
Omega ran a simulation of Newcomb’s problem, complete with a TDT agent in it. The simulated TDT agent obviously one boxed, and got the million. If you run TDT yourself, you also know it. Now, Omega tells you of this simulation, and tells you to chose your boxes. This is not Newcomb’s problem. If it was, deciding to 2 box would cause box B to be empty!
CDT would crudely assume that 2 boxing gets it $1000 more than 1 boxing. TDT on the other hand knows the simulated box B (and therefore the real one as well) has the million, regardless of its current decision.
###10 boxes problem
Again, the simulated problem and the real one aren’t the same. If there were, choosing box 1 with probability 1 would cause box 2 to have the million. Because it’s not the same problem, even TDT should be allowed to precommit different decision. The point of TDT is to foresee the consequences of its precommitments. It will therefore know that its precommitment in the real problem doesn’t have any influence to its precommitment (and therefore the outcome) in the simulated one. This lack of influence allows it to fall back on CDT reasoning.
Makes sense?
The simulated problem and the actual problem don’t have to actually be the same—just indistinguishable from the point of view of the agent.
Omega avoids infinite regress because the actual contents of the boxes are irrelevant for the purposes of the simulation, so no sub-simulation is necessary.
Okay. So, what specific mistake TDT does that would prevent it to distinguish the two problems? What does it lead it to think “If I precommit X in problem 1, I have to precommit X in problem 2 as well”.
(If the problems aren’t the same, of course Omega can avoid infinite regress. And if there is unbounded regress, we may be able to find a non-infinite solution by looping the regress over itself. But then the problems (simulated an real) are definitely the same.)
In the simulated problem the simulated agent is presented with the choice but never gets the reward; for all it matters both boxes can be empty. This means that Omega doesn’t have to do another simulation to work out what’s in the simulated boxes.
The infinite regress is resolvable anyway—since each TDT agent is facing the exact same problem, their decisions must be identical, hence TDT one-boxes and Omega knows this.
Of course.
Now there’s still the question of the perceived difference between the simulated problem and the real one (I assume here that you should 1 box in the simulation, and 2 box in the real problem). There is a difference, how come TDT does not see it? A Rational Decision Theory would —we humans do. Or if it can see it, how come can’t it act on it? RDT could. Do you concede that TDT does and can, or do you still have doubts?
Due to how the problem is set up, you can’t notice the difference until after you’ve made your decision. The only reason other decision theories know they’re not in the simulation is because the problem explicitly states that a TDT agent is simulated, which means it can’t be them.
That’s false. Here is a modified version of the problem:
Even if you’re not running TDT, the simulated agent is running the same decision algorithm as you are. If that was the reason why TDT couldn’t tell the difference, well, now no one can. However you and I can make the difference. The simulated problem is obviously different:
Really, the subjective difference between the two problems should be obvious to any remotely rational agent.
(Please let me know if you agree up until that point. Below, I assume you do.)
I’m pretty sure the correct answers for the two problems (my modified version as well as the original one) are 1-box in the simulation, 2-box in the real problem. (Do you still agree?)
So. We both agree that RDT (Rational Decision Theory) 1-boxes in the simulation, and 2-boxes in the real problem. CDT would 2-box in both, and TDT would 1-box in the simulation while in the real problem it would…
2-box? I think so.
1-box? Supposedly because it can’t tell simulation from reality. Or rather, it can’t tell the difference between Newcomb’s problem and the actual problem. Even though RDT does. (riiight?) So again, I must ask, why not? I need a more specific answer than “due to how the problem is set up”. I need you to tell me what specific kind of irrationality TDT is committing here. I need to know its specific blind spot.
In your problem, TDT does indeed 2-box, but it’s quite a different problem from the original one. Here’s the main difference:
vs
Oh. So this is indeed a case of “If you’re running TDT, I screw you, otherwise you walk free”. The way I understand this, TDT one boxes because it should.
If TDT cannot perceive the difference between the original problem and the simulation, this is because it is actually the same problem. For all it knows, it could be in the simulation (the simulation argument would say it is). There is an infinite regress, solved by the fact that all agents in all simulation level will have taken the same decision, because they ran the same decision algorithm. If they all 2-box, they get $1000, while if they all 1-box, they get the million (and no more).
Now, if you replace “an agent running TDT” by “you” (like, a fork of you started from a snapshot taken 3 seconds ago), the correct answer is always to 1-box, because then the problem is equivalent to the actual Newcomb’s problem.
Well, in the problem you present here TDT would 2-box, but you’ve avoided the hard part of the problem from the OP, in which there is no way to tell whether you’re in the simulation or not (or at least there is no way for the simulated you to tell), unless you’re running some algorithm other than TDT.
I see no such hard part.
To get back to the exact original problem as stated by the OP, I only need to replace “you” by “an agent running TDT”, and “your simulated twin” by “the simulated agent”. Do we agree?
Assuming we do agree, are you telling me the hard part is in that change? Are you telling me that TDT would 1-box in the original problem, even though it 2-boxes on my problem?
WHYYYYY?
Wait a minute, what exactly do you mean by “you”? TDT? or “any agent whatsoever”? If it’s TDT alone why? If I read you correctly, you already agree that’s it’s not because Omega said “running TDT” instead of “running WTF-DT”. If it’s “any agent whatsoever”, then are you really sure the simulated and real problem aren’t actually the same? (I’m sure they aren’t, but, just checking.)
Well, no, this would be my disagreement: it’s precisely because Omega told you that the simulated agent is running TDT that only TDT could or could not be the simulation; the simulated and real problem are, for all intents and purposes, identical (Omega doesn’t actually need to put a reward in the simulated boxes, because he doesn’t need to reward the simulated agent, but both problems appear exactly the same to the simulated and real TDT agents).
This comment from lackofcheese finally made it click. Your comment also make sense.
I now understand that this “problematic” problem just isn’t fair. TDT 1-boxes because it’s the only way to get the million.
Um, yes, they were. That’s the whole point.
I’ll need to write a full discussion post about that at some point. There is one crucial difference besides “I’m TDT” and “I’m CDT”. It’s “The simulated agent uses the same decision theory” and “The simulated agent does not use the same decision theory”.
That’s not exactly the same problem, and I think that is the whole point.