UDT has this same problem, though. In UDT, model uncertainty is being exploited instead of environmental uncertainty, but conditioning on “Agent takes action A” introduces spurious correlations with features of the model where it takes action A.
In particular, only one of the actions will happen in the models where con(PA) is true, so the rest of the actions occur in models where con(PA) is false, and this causes problems as detailed in “The Odd Counterfactuals of Playing Chicken” and the comments on “An Informal Conjecture on Proof Length and Logical Counterfactuals”.
I suspect this may also be relevant to non-optimality when the environment is proving things about the agent. The heart of doing well on those sorts of problems seems to be the agent trusting that the predictor will correctly predict its decision, but of course, a PA-based version of UDT can’t know that a PA or ZFC-based proof searcher will be sound regarding its own actions.
UDT has this same problem, though. In UDT, model uncertainty is being exploited instead of environmental uncertainty, but conditioning on “Agent takes action A” introduces spurious correlations with features of the model where it takes action A.
In particular, only one of the actions will happen in the models where con(PA) is true, so the rest of the actions occur in models where con(PA) is false, and this causes problems as detailed in “The Odd Counterfactuals of Playing Chicken” and the comments on “An Informal Conjecture on Proof Length and Logical Counterfactuals”.
I suspect this may also be relevant to non-optimality when the environment is proving things about the agent. The heart of doing well on those sorts of problems seems to be the agent trusting that the predictor will correctly predict its decision, but of course, a PA-based version of UDT can’t know that a PA or ZFC-based proof searcher will be sound regarding its own actions.