“Coin chosen adversarially” is not just a more complex state space or a hierarchical hyperprior, it is an exit from probability entirely. (It is “demonic nondeterminism”, which is a form of uncertainty that is not expressible via probability.)
Functional Decision Theory and Logical Decision Theory, although obviously the right sort of direction, have never been given proper formal definitions. I believe that part of the reason for this is that to do so requires a fundamentally nonprobabilistic notion of belief state. For example, Logical Inductors, which were a step toward this, have a notion of belief state which is nonprobabilistic (I think it is a kind of partial prevision, which assigns to some gambles a price, without demanding these prices be complete or obey the laws of probability proper).
“Coin chosen adversarially” is an element of a state space. The agent assigns a probability to it being in different possible worlds.
JBlack and I do not seem to have made arguments that depend on the agent’s decision theory. (For all we’ve said, it might be ignoring the world model and acting randomly.) We have only discussed probability distributions that admit representing the Omega situation you brought up.
You would likely want to have your agent able to consider multiple different possible worlds, and evaluate the probability of being in any one of them. The typical highly general formalization of this is Solomonoff induction.
Even if you use the full machinery of Solomonoff induction, or any other process for Bayesian inference over large hypothesis classes, there is still a marginal probability distribution for the outcome of the coin, and therefore the d20 bet remains always strictly dominated by the bet on Heads, or on Tails, or both, regardless of how large the joint distribution was.
You have a few options for where to sample the predictions.
I think the best one is p(outcome | prospective agent action). i.e. consider what would happen if you take an action.
You could compute p(outcome), implicitly assuming that the agent’s actions will not impact the outcome it is about to observe. This will often be false.
You could also compute p(outcome) assuming that the agent follows it’s existing policy. This kind of self-prediction seems likely to lead to some strange situations, and also assumes that the policy has already decided on a course of action before we sample the prediction about the outcome (in which case why are we calculating probabilities of outcomes, if they can’t inform the prediction?)
I think that the purpose of modelling the world is mostly to answer questions of the form “what happens if I do X?”, and that form 1 works best. If we use form 1, then there will be a distribution for p(Heads | Agent chooses to bet on heads) and p(Tails | Agent chooses to bet on tails). A truing machine is perfectly capable of outputting 0 for both of those (or any other arbitrary number). If the inference process has incorporated the fairness of the d20, it will return p(d20 > 12 | Agent bets on d20) = 8⁄20.
If you use 2, you end up with an agent that can’t handle Newcomblike situations. More severely, you end up with an agent that does not know “If I drive into that building, something bad will happen”, as it is ignoring its action in the prediction. 3 probably depends on how you handle the self reference.
“Coin chosen adversarially” is not just a more complex state space or a hierarchical hyperprior, it is an exit from probability entirely. (It is “demonic nondeterminism”, which is a form of uncertainty that is not expressible via probability.)
Functional Decision Theory and Logical Decision Theory, although obviously the right sort of direction, have never been given proper formal definitions. I believe that part of the reason for this is that to do so requires a fundamentally nonprobabilistic notion of belief state. For example, Logical Inductors, which were a step toward this, have a notion of belief state which is nonprobabilistic (I think it is a kind of partial prevision, which assigns to some gambles a price, without demanding these prices be complete or obey the laws of probability proper).
“Coin chosen adversarially” is an element of a state space. The agent assigns a probability to it being in different possible worlds.
JBlack and I do not seem to have made arguments that depend on the agent’s decision theory. (For all we’ve said, it might be ignoring the world model and acting randomly.) We have only discussed probability distributions that admit representing the Omega situation you brought up.
You would likely want to have your agent able to consider multiple different possible worlds, and evaluate the probability of being in any one of them. The typical highly general formalization of this is Solomonoff induction.
Even if you use the full machinery of Solomonoff induction, or any other process for Bayesian inference over large hypothesis classes, there is still a marginal probability distribution for the outcome of the coin, and therefore the d20 bet remains always strictly dominated by the bet on Heads, or on Tails, or both, regardless of how large the joint distribution was.
You have a few options for where to sample the predictions.
I think the best one is p(outcome | prospective agent action). i.e. consider what would happen if you take an action.
You could compute p(outcome), implicitly assuming that the agent’s actions will not impact the outcome it is about to observe. This will often be false.
You could also compute p(outcome) assuming that the agent follows it’s existing policy. This kind of self-prediction seems likely to lead to some strange situations, and also assumes that the policy has already decided on a course of action before we sample the prediction about the outcome (in which case why are we calculating probabilities of outcomes, if they can’t inform the prediction?)
I think that the purpose of modelling the world is mostly to answer questions of the form “what happens if I do X?”, and that form 1 works best. If we use form 1, then there will be a distribution for p(Heads | Agent chooses to bet on heads) and p(Tails | Agent chooses to bet on tails). A truing machine is perfectly capable of outputting 0 for both of those (or any other arbitrary number). If the inference process has incorporated the fairness of the d20, it will return p(d20 > 12 | Agent bets on d20) = 8⁄20.
If you use 2, you end up with an agent that can’t handle Newcomblike situations. More severely, you end up with an agent that does not know “If I drive into that building, something bad will happen”, as it is ignoring its action in the prediction. 3 probably depends on how you handle the self reference.