Hard perhaps does not cut it. If this thing self-modifies more than once (as is expected), you run into an at least exponential explosion in resource use, growing to planet-eating sizes even before anything useful gets done. If you don’t take into account further self-modifications, how can you claim that you chose the best? And then there are the problems of simulating the next 10 billion years with any accuracy...
But on thinking about it I’m not sure why the decision theory is particularly difficult either. Maybe if you wanted to not just use higher-level properties of the modification and instead be able to modify “on the fly,” e.g. if Omega says he’ll give you 10 dollars if you choose box A by the process of choosing alphabetically. There might also be some interesting problems in bargaining with an agent to change its terminal values.
Hard perhaps does not cut it. If this thing self-modifies more than once (as is expected), you run into an at least exponential explosion in resource use, growing to planet-eating sizes even before anything useful gets done. If you don’t take into account further self-modifications, how can you claim that you chose the best? And then there are the problems of simulating the next 10 billion years with any accuracy...
But on thinking about it I’m not sure why the decision theory is particularly difficult either. Maybe if you wanted to not just use higher-level properties of the modification and instead be able to modify “on the fly,” e.g. if Omega says he’ll give you 10 dollars if you choose box A by the process of choosing alphabetically. There might also be some interesting problems in bargaining with an agent to change its terminal values.
Is the purpose of CDT/UDT/TDT is to arrive at a computationally efficient decision procedure? I’ve never seen this stated explicitly.
No.
Nope. :D