“Coin chosen adversarially” is an element of a state space. The agent assigns a probability to it being in different possible worlds.
JBlack and I do not seem to have made arguments that depend on the agent’s decision theory. (For all we’ve said, it might be ignoring the world model and acting randomly.) We have only discussed probability distributions that admit representing the Omega situation you brought up.
You would likely want to have your agent able to consider multiple different possible worlds, and evaluate the probability of being in any one of them. The typical highly general formalization of this is Solomonoff induction.
Even if you use the full machinery of Solomonoff induction, or any other process for Bayesian inference over large hypothesis classes, there is still a marginal probability distribution for the outcome of the coin, and therefore the d20 bet remains always strictly dominated by the bet on Heads, or on Tails, or both, regardless of how large the joint distribution was.
You have a few options for where to sample the predictions.
I think the best one is p(outcome | prospective agent action). i.e. consider what would happen if you take an action.
You could compute p(outcome), implicitly assuming that the agent’s actions will not impact the outcome it is about to observe. This will often be false.
You could also compute p(outcome) assuming that the agent follows it’s existing policy. This kind of self-prediction seems likely to lead to some strange situations, and also assumes that the policy has already decided on a course of action before we sample the prediction about the outcome (in which case why are we calculating probabilities of outcomes, if they can’t inform the prediction?)
I think that the purpose of modelling the world is mostly to answer questions of the form “what happens if I do X?”, and that form 1 works best. If we use form 1, then there will be a distribution for p(Heads | Agent chooses to bet on heads) and p(Tails | Agent chooses to bet on tails). A truing machine is perfectly capable of outputting 0 for both of those (or any other arbitrary number). If the inference process has incorporated the fairness of the d20, it will return p(d20 > 12 | Agent bets on d20) = 8⁄20.
If you use 2, you end up with an agent that can’t handle Newcomblike situations. More severely, you end up with an agent that does not know “If I drive into that building, something bad will happen”, as it is ignoring its action in the prediction. 3 probably depends on how you handle the self reference.
“Coin chosen adversarially” is an element of a state space. The agent assigns a probability to it being in different possible worlds.
JBlack and I do not seem to have made arguments that depend on the agent’s decision theory. (For all we’ve said, it might be ignoring the world model and acting randomly.) We have only discussed probability distributions that admit representing the Omega situation you brought up.
You would likely want to have your agent able to consider multiple different possible worlds, and evaluate the probability of being in any one of them. The typical highly general formalization of this is Solomonoff induction.
Even if you use the full machinery of Solomonoff induction, or any other process for Bayesian inference over large hypothesis classes, there is still a marginal probability distribution for the outcome of the coin, and therefore the d20 bet remains always strictly dominated by the bet on Heads, or on Tails, or both, regardless of how large the joint distribution was.
You have a few options for where to sample the predictions.
I think the best one is p(outcome | prospective agent action). i.e. consider what would happen if you take an action.
You could compute p(outcome), implicitly assuming that the agent’s actions will not impact the outcome it is about to observe. This will often be false.
You could also compute p(outcome) assuming that the agent follows it’s existing policy. This kind of self-prediction seems likely to lead to some strange situations, and also assumes that the policy has already decided on a course of action before we sample the prediction about the outcome (in which case why are we calculating probabilities of outcomes, if they can’t inform the prediction?)
I think that the purpose of modelling the world is mostly to answer questions of the form “what happens if I do X?”, and that form 1 works best. If we use form 1, then there will be a distribution for p(Heads | Agent chooses to bet on heads) and p(Tails | Agent chooses to bet on tails). A truing machine is perfectly capable of outputting 0 for both of those (or any other arbitrary number). If the inference process has incorporated the fairness of the d20, it will return p(d20 > 12 | Agent bets on d20) = 8⁄20.
If you use 2, you end up with an agent that can’t handle Newcomblike situations. More severely, you end up with an agent that does not know “If I drive into that building, something bad will happen”, as it is ignoring its action in the prediction. 3 probably depends on how you handle the self reference.