I usually just think about which decision theory we’d want to program into an AI which might get copied, its source code inspected, etc. That lets you get past the basic stuff, like Newcomb’s Problem, and move on to more interesting things. Then you can see which intuitions can be transferred back to problems involving humans.
It turns out that many of the complications (multiple players, amnesia, copying, predictors, counterfactuals) lead to the same idea: that we should model things game-theoretically and play the global optimal strategy no matter what, instead of trying to find the optimal decision locally at each node. That idea summarizes a large part of UDT (Wei’s originalproposals of UDT also included dealing with logical uncertainty, but that turned out to be much harder.) Hence my recent posts on how to model anthropic updates and predictors game-theoretically.
“I usually just think about which decision theory we’d want to program into an AI which might get copied, its source code inspected”—well everyone agrees that if you can pre-commit to one-box you ought to. The question is what about if you’re in the situation and you haven’t pre-committed. My answer is that if you take a choice, then you were implicitly pre-committed.
People who follow UDT don’t need to precommit, they have a perfectly local decision procedure: think back, figure out the best strategy, and play a part in it. The question of precommitment only arises if you follow CDT, but why would you follow CDT?
Correct, but you’re justifying UDT by arguing what you should do if you had pre-committed. A two-boxer would argue that this is incorrect because you haven’t pre-committed.
The idea of playing the best strategy can stand on its own, it doesn’t need to be justified by precommitment. I’d say the idea of myopically choosing the next move needs justification more.
For example, when you’re dealt a weak hand in poker, the temptation to fold is strong. But all good players know you must play aggressively on your weakest hands, because if you fold, you might as well light up a neon sign saying “I have a strong hand” whenever you do play aggressively, allowing your opponent to fold and cut their losses. In this case it’s clear that playing the best strategy is right, and myopically choosing the next move is wrong. You don’t need precommitment to figure it out. Sure, it’s a repeated game where your opponent can learn about you, but Newcomb’s Problem has a predictor which amounts to the same thing.
I usually just think about which decision theory we’d want to program into an AI which might get copied, its source code inspected, etc. That lets you get past the basic stuff, like Newcomb’s Problem, and move on to more interesting things. Then you can see which intuitions can be transferred back to problems involving humans.
It turns out that many of the complications (multiple players, amnesia, copying, predictors, counterfactuals) lead to the same idea: that we should model things game-theoretically and play the global optimal strategy no matter what, instead of trying to find the optimal decision locally at each node. That idea summarizes a large part of UDT (Wei’s original proposals of UDT also included dealing with logical uncertainty, but that turned out to be much harder.) Hence my recent posts on how to model anthropic updates and predictors game-theoretically.
“I usually just think about which decision theory we’d want to program into an AI which might get copied, its source code inspected”—well everyone agrees that if you can pre-commit to one-box you ought to. The question is what about if you’re in the situation and you haven’t pre-committed. My answer is that if you take a choice, then you were implicitly pre-committed.
People who follow UDT don’t need to precommit, they have a perfectly local decision procedure: think back, figure out the best strategy, and play a part in it. The question of precommitment only arises if you follow CDT, but why would you follow CDT?
Correct, but you’re justifying UDT by arguing what you should do if you had pre-committed. A two-boxer would argue that this is incorrect because you haven’t pre-committed.
The idea of playing the best strategy can stand on its own, it doesn’t need to be justified by precommitment. I’d say the idea of myopically choosing the next move needs justification more.
For example, when you’re dealt a weak hand in poker, the temptation to fold is strong. But all good players know you must play aggressively on your weakest hands, because if you fold, you might as well light up a neon sign saying “I have a strong hand” whenever you do play aggressively, allowing your opponent to fold and cut their losses. In this case it’s clear that playing the best strategy is right, and myopically choosing the next move is wrong. You don’t need precommitment to figure it out. Sure, it’s a repeated game where your opponent can learn about you, but Newcomb’s Problem has a predictor which amounts to the same thing.