Basically, all the local decisions come from the same computation that would be performed to set the most general precommitment for all possible states of the world. The expected utility maximization is defined only once, on the global state space, and then the actual actions only retrieve the global solution, given encountered observations. The observations don’t change the state space over which the expected utility optimization is defined (and don’t change the optimal global solution or preference order on the global solutions), only what the decisions in a given (counterfactual) branch can affect. Since the global precommitment is the only thing that defines the local agents’ decisions, the “commitment” part can be dropped, and the agents’ actions can just be defined to follow the resulting preference order.
In this solution, there is no belief updating; there is just decision theory. (All probabilities are “timestamped” to the beliefs of the agent’s creator when the agent was created.) This means that the use of Bayesian belief updating with expected utility maximization may be just an approximation that is only relevant in special situations which meet certain independence assumptions around the agent’s actions. In the more general Newcomb-like family of situations, computationally efficient decision algorithms might use a family of approximations more general than Bayesian updating.
There would, for example, be no such thing as “posterior probability of ‘coin comes up heads’” or “probability that you are a Boltzmann brain”; there would only be a fraction of importance-measure that brains with your decision algorithm could affect. As Vladimir Nesov commented:
Agents self-consistent under reflection are counterfactual zombies, indifferent to whether they are real or not.
Anna and I noticed this possible decision rule around four months before Vladimir posted it (with “possible observations” replaced by “partial histories of sense data and actions”, and also some implications about how to use limited computing power on “only what the decisions in a given (counterfactual) branch can affect” while still computing predicted decisions on one’s other counterfactual branches well enough to coordinate with them). But we didn’t write it up to a polished state, partly because we didn’t think it seemed enough like it was the central insight in the area. Mostly, that was because this decision rule doesn’t explain how to think about any logical paradoxes of self-reference, such as algorithms that refer to each others’ output. It also doesn’t explain how to think about logical uncertainty, such as the parity of the trillionth digit of pi, because the policy optimization is assumed to be logically omniscient. But maybe we were wrong about how central it was.
It looks like the uncertainty about your own actions in other possible worlds is entirely analogous to uncertainty about mathematical facts: in both cases, the answer is in denotation of the structure you already have at hand, so it doesn’t seem like the question about your own actions should be treated differently from any other logical question.
(The following is moderately raw material and runs a risk of being nonsense, I don’t understand it well enough.)
One perspective that wasn’t mentioned and that I suspect may be important is considering interaction between different processes (or agents) as working by the same mechanism as common partial histories between alternative versions of the same agent. If you can have logical knowledge about your own actions in other possible states that grow in time and possibilities from your current structure, the same treatment can be given to possible states of the signal you send out, in either time-direction, that is to consequences of actions and observations. One step further, any knowledge (properly defined) you have at all about something else gives the same power of mutual coordination with that something, as the common partial history gives to alternative or at-different-times versions of yourself.
This problem seems deeply connected to logic and theoretical computer science, in particular models of concurrency.
By the way, you say “partial histories of sense data and actions”. I try considering this problem in time-reversible dynamic, it adds a lot of elegance, and there actions are not part of history, but more like something that is removed from history. State of the agent doesn’t accumulate from actions and observations, instead it’s added to by observations and taken away from by actions. The point at which something is considered observation or action and not part of agent’s state is itself rather arbitrary, and both can be seen as points of shifting the scope on what is considered part of agent. (This doesn’t have anything agent-specific, and is more about processes in general.)
Everything you said sounds correct, except the last bit, which is just unclear to me. I’d welcome a demonstration (or formal definition) some day:
By the way, you say “partial histories of sense data and actions”. I try considering this problem in time-reversible dynamic, it adds a lot of elegance, and there actions are not part of history, but more like something that is removed from history. State of the agent doesn’t accumulate from actions and observations, instead it’s added to by observations and taken away from by actions. The point at which something is considered observation or action and not part of agent’s state is itself rather arbitrary, and both can be seen as points of shifting the scope on what is considered part of agent. (This doesn’t have anything agent-specific, and is more about processes in general.)
I thought the answer Vladimir Nesov already posted solved Counterfactual Mugging for a quantum coin?
In this solution, there is no belief updating; there is just decision theory. (All probabilities are “timestamped” to the beliefs of the agent’s creator when the agent was created.) This means that the use of Bayesian belief updating with expected utility maximization may be just an approximation that is only relevant in special situations which meet certain independence assumptions around the agent’s actions. In the more general Newcomb-like family of situations, computationally efficient decision algorithms might use a family of approximations more general than Bayesian updating.
There would, for example, be no such thing as “posterior probability of ‘coin comes up heads’” or “probability that you are a Boltzmann brain”; there would only be a fraction of importance-measure that brains with your decision algorithm could affect. As Vladimir Nesov commented:
Anna and I noticed this possible decision rule around four months before Vladimir posted it (with “possible observations” replaced by “partial histories of sense data and actions”, and also some implications about how to use limited computing power on “only what the decisions in a given (counterfactual) branch can affect” while still computing predicted decisions on one’s other counterfactual branches well enough to coordinate with them). But we didn’t write it up to a polished state, partly because we didn’t think it seemed enough like it was the central insight in the area. Mostly, that was because this decision rule doesn’t explain how to think about any logical paradoxes of self-reference, such as algorithms that refer to each others’ output. It also doesn’t explain how to think about logical uncertainty, such as the parity of the trillionth digit of pi, because the policy optimization is assumed to be logically omniscient. But maybe we were wrong about how central it was.
It looks like the uncertainty about your own actions in other possible worlds is entirely analogous to uncertainty about mathematical facts: in both cases, the answer is in denotation of the structure you already have at hand, so it doesn’t seem like the question about your own actions should be treated differently from any other logical question.
(The following is moderately raw material and runs a risk of being nonsense, I don’t understand it well enough.)
One perspective that wasn’t mentioned and that I suspect may be important is considering interaction between different processes (or agents) as working by the same mechanism as common partial histories between alternative versions of the same agent. If you can have logical knowledge about your own actions in other possible states that grow in time and possibilities from your current structure, the same treatment can be given to possible states of the signal you send out, in either time-direction, that is to consequences of actions and observations. One step further, any knowledge (properly defined) you have at all about something else gives the same power of mutual coordination with that something, as the common partial history gives to alternative or at-different-times versions of yourself.
This problem seems deeply connected to logic and theoretical computer science, in particular models of concurrency.
By the way, you say “partial histories of sense data and actions”. I try considering this problem in time-reversible dynamic, it adds a lot of elegance, and there actions are not part of history, but more like something that is removed from history. State of the agent doesn’t accumulate from actions and observations, instead it’s added to by observations and taken away from by actions. The point at which something is considered observation or action and not part of agent’s state is itself rather arbitrary, and both can be seen as points of shifting the scope on what is considered part of agent. (This doesn’t have anything agent-specific, and is more about processes in general.)
Everything you said sounds correct, except the last bit, which is just unclear to me. I’d welcome a demonstration (or formal definition) some day:
Just curious, did you get the name “ambient control” from ambient calculi?
(It’s strange that I can use the language of possibility like that!)