Seems to me that it ought be treatable as “perfectly ordinary”...
That is, if you run a simulation, there’s no reason to for you to believe the same things that the modeled beings believe, right? If one of the modeled beings happen to be a version of you that’s acting and believing in terms of a counterfactual that is the premise of the simulation, then… why would that automatically lead to you believing the same thing in the first place? If you simulate a piece of paper that has written upon it “1+1=3”, does that mean that you actually believe “1+1=3″? So if instead you simulate a version of yourself that gets confused and believes that “1+1=3”… well, that’s just a simulation. If there’s a risk of that escalating into your actual model of reality, that would suggest something is very wrong somewhere in how you set up a simulation in the first place, right?
ie, simulated you is allowed to make all the usual inferences from, well, other stuff in the simulated world. It’s just that actual you doesn’t get to automatically equate simulated you’s beliefs with actual you’s beliefs.
So allow the simulated version to make all the usual inferences. I don’t see why any restriction is needed other than the level separation, which doesn’t need to treat this issue as a special case.
ie, simulated you in the counterfactual in which A6 was chosen believes that, well, A6 is what the algorithm in question would choose as the best choice. So? You calmly observe/model the actions simulated you takes if it believes that and so on without having to actually believe that yourself. Then, once all the counterfactual modelings are done and you apply your utility function to each of those to determine their actual expected utility, thus finding that A7 produces the highest EU, you actually do A7.
It simply happens to be that most of the versions of you from the counterfactual models that arose in the process of doing the TDT computation had false beliefs about what the actual output of the computation actually is in actual reality.
Am I missing the point still, or...?
(wait… I’m understanding this issue to be something that you consider an unsolved issue in TDT and I’m saying “no, seems to me to be simple to make TDT do the right thing here. The Pearl style counterfactual stuff oughtn’t cause any problem here, no special cases, no forbidden inferences need to be hard coded here”, but now, looking at your comment, maybe you meant “This issue justifies TDT because TDT actually does the right thing here”, in which case there was no need for me to say any of this at all. :))
Seems to me that it ought be treatable as “perfectly ordinary”...
That is, if you run a simulation, there’s no reason to for you to believe the same things that the modeled beings believe, right? If one of the modeled beings happen to be a version of you that’s acting and believing in terms of a counterfactual that is the premise of the simulation, then… why would that automatically lead to you believing the same thing in the first place? If you simulate a piece of paper that has written upon it “1+1=3”, does that mean that you actually believe “1+1=3″? So if instead you simulate a version of yourself that gets confused and believes that “1+1=3”… well, that’s just a simulation. If there’s a risk of that escalating into your actual model of reality, that would suggest something is very wrong somewhere in how you set up a simulation in the first place, right?
ie, simulated you is allowed to make all the usual inferences from, well, other stuff in the simulated world. It’s just that actual you doesn’t get to automatically equate simulated you’s beliefs with actual you’s beliefs.
So allow the simulated version to make all the usual inferences. I don’t see why any restriction is needed other than the level separation, which doesn’t need to treat this issue as a special case.
ie, simulated you in the counterfactual in which A6 was chosen believes that, well, A6 is what the algorithm in question would choose as the best choice. So? You calmly observe/model the actions simulated you takes if it believes that and so on without having to actually believe that yourself. Then, once all the counterfactual modelings are done and you apply your utility function to each of those to determine their actual expected utility, thus finding that A7 produces the highest EU, you actually do A7.
It simply happens to be that most of the versions of you from the counterfactual models that arose in the process of doing the TDT computation had false beliefs about what the actual output of the computation actually is in actual reality.
Am I missing the point still, or...?
(wait… I’m understanding this issue to be something that you consider an unsolved issue in TDT and I’m saying “no, seems to me to be simple to make TDT do the right thing here. The Pearl style counterfactual stuff oughtn’t cause any problem here, no special cases, no forbidden inferences need to be hard coded here”, but now, looking at your comment, maybe you meant “This issue justifies TDT because TDT actually does the right thing here”, in which case there was no need for me to say any of this at all. :))