Vanessa Kosoy comments on Vanessa Kosoy’s Shortform

[Vanessa Kosoy]

This is an idea I came up with and presented in the Agent Foundations 2025 at CMU conference.

Here is a nice simple formalism for decision theory, that in particular supports the decision theory coming out of infra-Bayesianism. I now call the latter decision theory “Disambiguative Decision Theory”, since the counterfactuals work by “disambiguating” the agent’s belief.

Formalism

Let be the agent’s event space and the space of possible policies ^[1] . Let be the agent’s loss function. For each , we are given some . ^[2] This represents the event “the agent’s behavior is consistent with policy ”. We assume that

This data is common for all decision theories, but the rest of the details depend on the theory:

Functional Decision Theory (FDT)

We are given a mapping s.t. is supported on for all . The distribution represents the logical counterfactual associated with . It is also possible to consider the more general “robust” version , but we will avoid it here for simplicity. The decision rule is then

We will call an FDT problem “formally causal” when for any , the measures and agree when restricted to . That is, for any measurable , we require

Causal Decision Theory (CDT)

CDT has the same formal form as FDT, but we always require the problem to formally causal. Moreover, the interpretation of is different: it now represents the causal counterfactual associated with . The decision rule is also formally the same:

Given an FDT problem , we can translate it to a CDT problem, if we specify the agent’s belief about its own policies and causal interpretation: the kernel . Here is a copy of that represents the factual policy and is a copy of that represents the counterfactual policy. We require that , and that is formally causal in the second argument.

Given this data, we define the translation

Extensive Form and Evidential Decision Theory (EDT)

Extensive Form

To formalize EDT, we need to assume the decision process is given in “extensive” form. That is, we have a set of decision points, for each a set of actions , and a mapping , that defines the previous decision point and action. Here, we use the notation

We assume that is acyclic and hence makes into the vertices of a forest whose edges are labeled by .

We define a policy to be s.t.

2. For every , there is at most one s.t. .

3. For every , if then there exists some s.t. .

is now the set of policies defined in this way.

We further assume that there is a mapping (representing the last action taken) s.t. for all

Here, stands for iterating in the obvious way.

For any , we can use the notation

This represents the event “the decision point actually takes place”.

EDT

So far, this notion of extensive form decision problem is useful not just for EDT. Specifically for EDT, we add the assumption that we’re given the agent’s belief . We can now state the EDT decision rule. We define recursively. Always, .

For every s.t. , we set

Thus, the agent conditions both on following policy and observing decision point .

Given an FDT problem in extensive form, we can translate it to a EDT problem, if we specify the agent’s belief about its own policies . We define the translation

Disambiguative Decision Theory (DDT)

We are given the agent’s belief . Here, refers to supracontributions. The decision rule is

Here, is the characteristic function of the set . Equivalently, we can define by

We then have

This is the reason for the name “disambiguative”: is a “disambiguated” version of , where the policy is made unambiguous.

Given an FDT problem , we can translate it to a DDT problem without any further data:

That is, is the supracontribution hull of the distributions when ranges over .

DDT does have the odd property of non-invariance w.r.t. shifting by a constant, as opposed to all other decision theories considered. There might be some story about how this non-invariance is an inevitable consequence of learning (where imposing bounds on is important), but I’m not ready to tell it.

Comparison

Now, let’s look into how different decision theories compare. We will be using FDT as the “gold standard” throughout, when it comes to choosing the correct policy. Note though, that FDT assumes we somehow assign strict meaning to the logical counterfactuals, which is unclear how to accomplish. On the other hand, DDT makes the substantially weaker assumption that can define the supracontribution belief. In particular, it is consistent with learning, as was explained here.

Proposition 1: Consider a formally causal FDT problem . Assume that the causal interpretation takes the form . Then, .

Proposition 2: Consider a formally causal FDT problem in extensive form. Then, .

Proposition 3: Consider a formally causal FDT problem. Then, Then, .

Thus, in the strictly causal case all decision theories coincide: but even here DDT requires the least precise assumptions for that to work (compared to CDT and EDT). More importantly, DDT allows to go far beyond the formally causal case. However, we do need a mild assumption about the problem:

Definition 1: An FDT problem is called pseudocausal when for any , if then .

It’s easy to see that any formally causal problem is pseudocausal, but there are many counterexamples to the converse.

Essentially, pseudocausality means that the outcome cannot depend on decisions in situations of probability 0. Notice that in reality the agent is never absolutely certain about the decision problem, hence observing a situation of probability 0 should cause it to believe it is in a different decision problem altogether. This makes the pseudocausality condition very natural.

Pseudocausality has the nice property of not depending on the loss function. If we do allow dependence on the loss function, we can make do with an even weaker condition.

Definition 2: An FDT problem is called stable when there exists an FDT-optimal s.t. for any , if then is also FDT-optimal.

It’s obvious that any stable problem is pseudocausal. Naturally, the converse is false.

Neither pseudocausality nor stability is sufficient to guarantee that DDT and FDT give identical recommendations. However, it becomes true when we iterate the problem.

Definition 3: Given a decision problem and , we define its -th power as follows. The event space is just the ordinary power . The policy space is . The loss function is

Given , we define by

For FDT, for any we define the kernel by . We then define the logical counterfactuals

For DDT, we take the belief to be .

---

Note that iterating a problem commutes with converting it from FDT to DDT.

Theorem 4: For a stable FDT problem, there exists s.t. for any , DDT and FDT agree on the problem .

The requirement to iterate doesn’t seem like a terrible cost, since in a learning context some kind of iteration is necessary anyway. It can also be understood as a natural result of the need for stability: problems that are close to being unstable require more iterations.

Examples

All these examples besides the last one have natural extensive forms with one decision point.

Newcomb

This problem is formally causal, however the usual causal interpretation is non-trivial:

As a result, .

XOR Blackmail

The problem is pseudocausal but not formally causal. Nevertheless, CDT agrees with FDT thanks to the following causal interpretation:

Counterfactual Mugging

The problem is pseudocausal but not formally causal.

Empty-Dependent Transparent Newcomb

For simplicity, we postulate that the agent is forced to two-box when seeing a full box, since this choice is a “no-brainer” for all decision theories.

The problem is stable but not pseudocausal. EDT is ill-posed because , where is the unique decision point (that corresponds to seeing an empty box).

Full-Dependent Transparent Newcomb

As above, we postulate that the agent is forced to two-box when seeing an empty box.

The problem is not stable. EDT is ill-posed because , where is the unique decision point (that corresponds to seeing an full box). DDT is indifferent between and , but it’s possible to construct a variant where DDT is strictly FDT-suboptimal.

Full-Dependent Transparent Newcomb with Noise

We now assume Omega has a probability of filling the box even when the agent two-boxes.

The problem is pseudocausal, but not formally causal of course. EDT is well-posed and . DDT converges to FDT after iterations.

Self-Coordination

Here’s an interesting example of a problem with two decision points. Omega flips a coin and shows the result to the agent. The agent then has to choose between buttons A, B and C. Button C always yields 3 dollars. Buttons A and B yield 4 dollars if Omega predicts the agent would choose the same button in the other coin counterfactual, and 0 dollars otherwise.

The rest of the definitions are clear and we won’t write them out. The problem is pseudocausal but not formally causal. CDT and EDT agree here, with their behavior depending on the agent’s self-belief . For uniform they choose the FDT-suboptimal policy . Moreover, there is an “equilibrium” where they choose even for “calibrated” (i.e. that puts most of the probability mass on ).

1. ↩︎

   It is simplest to think of both as finite sets, but they can also be compact Polish spaces.

2. ↩︎

   In the topological case, is required to be closed.

[Vanessa Kosoy]

A few more observations.

Partially Observable Iteration

The definition of iteration we had before implicitly assumes that the agent can observe the full outcome of previous iterations. We don’t have to make this assumption. Instead, we can assume a set of possible observations and a mapping , in which case we define

I believe that Theorem 4 remains valid.

Idealized Disambiguative Decision Theory

As we remarked before, DDT is not invariant under adding a constant to the loss function. It is interesting to consider what happens when we add an increasingly large constant. In the limit, DDT converges to something I dubbed “Idealized Disambiguative Decision Theory” (IDDT)^[1], which works as follows.

For IDDT, it is sufficient to let be crisp (i.e. a credal set). We may allow supracontributions if we wish, but any problem defined via “unambiguous” FDT (i.e. as opposed to ) reduces to the crisp case. Define by

For problems coming from unambiguous FDT, , but IDDT is defined in full generality. For every , define by

The decision rule is then

Notice that it is now invariant w.r.t. adding constants to . Moreover,

Proposition 5: For any stable problem, it holds that (i) any IDDT-optimal policy is FDT-optimal (ii) there is an FDT-optimal policy which is IDDT-optimal. For any pseudocausal problem, it also holds that any FDT-optimal policy is IDDT-optimal.

One might think, based on this proposition, that IDDT is a superior decision theory to DDT. However, I think that IDDT is incompatible with learning, because of its discontinuous dependence on probabilities.

More Examples

Absent-Minded Driver

(Based on Aumann, Hart and Perry.) We will operationalize the problem by assuming the agent’s decision may deterministically depend on observing a coin flip. To simplify the presentation, we assume a single coin flip per intersection, which limits the resulting probabilities to , but it’s easy to generalize further.

Denote by and the constant policies. Denote by the policy

Denote by the remaining policy.

Consistently with our source, we set the loss function to be , , (it doesn’t depend on the coin flips).

This problem is formally causal. However, as opposed to all previous examples, it has no extensive form! Hence, EDT in the sense we defined it is ill-posed: to apply EDT reasoning here we need to at least supplement it by a theory of anthropic probabilities. CDT’s counterfactuals agree with FDT’s if we posit that the do-operator is constrained to choosing among “absent-minded” policies.

Self-Prisoner’s Dilemma

Previously we described the self-coordination problem, but perhaps self-PD is a more striking example.

Here, and is the agent’s factual play, whereas and is the agent’s counterfactual play as predicted by Omega.

Using the obvious notations , we have

The loss is the usual PD loss of the “factual” player.

This problem is not formally causal, because e.g.

The natural CDT interpretation is the one where the factual policy controls the counterfacual player and the counterfactual policy controls the factual player. (Alas, the terminology gets confusing here: in one case the words “factual” and “counterfactual” refer to the agent’s policy, and in the other case to the coin’s outcome.) Both CDT and EDT play regardless of self-belief. However, the problem is pseudocausal and hence DDT converges to .

1. ^

   IDDT is related to the old idea of “surmeasures” from the original infra-Bayesianism sequence.

2. ^

   We can also imagine equipping the agent with a “self-belief” (not necessarily ) and setting , in which case also becomes relevant.

[Vanessa Kosoy]

Above, I compare different decision theories to FDT. At the same time, I claim that in a deeper sense, FDT is ill-defined. One may doubt whether that is a coherent line of reasoning. Therefore, instead of a comparison to FDT, I propose to frame these observations as being about stability to precommitments. Details follow.

Definition: A precommitment game is a decision problem in which
1. We are given some (the precommitment policies). We will denote .
2. We are given : for each precommitment , it says to which policy this is precomitting.
3. For any , if then : if a precommitting decision is made, then it is the only decision.
4. Denote , . (Notice that .) We are given a relation s.t. (i) implies and (ii) and implies . This tells us which precommitted outcomes correspond to which unprecomitted outcomes.

The restriction of to is called the underlying decision problem of .

Definition: An EDT precommitment game is a precommitment game which is also an EDT problem (i.e. extensive form and equipped with ) s.t. the following property holds. Denote (the “external” i.e unprecomittable policies) and . We require that there is some and s.t.
1. is a convex combination of and .^[1]
2. is in the convex hull of where ranges over .

The underlying decision problem is then an EDT decision problem with the belief .

as above with is called formally causal when for any and

(Is there a natural generalization without the assumption ? I don’t know.)

Proposition: EDT is precommitment-stable in formally causal precommitment games. That is, in any such game there is which is EDT-optimal.

For example, XOR blackmail can be formalized as an EDT precommitment game which is not formally causal and EDT is not precommitment-stable there (the only optimal policy is precommitting to reject).

Definition: [EDIT: The treatment of CDT here is problematic, see child post.] A CDT precommitment game is a precommitment game which is also a CDT problem (in the sense that we are given and formally causal in the second argument) s.t. the following property holds. For any and , there is some s.t. and .

The underlying decision problem is then a CDT decision problem with and .

as above is called policy-bottlenecked when for any , . (Because this condition holds when the decision nodes form a bottleneck in the causal network s.t. the outcome depends only on nodes on the downstream side.)

Proposition: CDT is precommitment-stable in policy-bottlenecked precommitment games. That is, in any such game there is which is CDT-optimal.

For example, Newcomb’s paradox can be formalized as a CDT precommitment game is which is not policy-bottlenecked and CDT is not precommitment-stable there (the only optimal policy is precommitting to one-box).

Definition: A DDT precommitment game is a precommitment game which is also a DDT problem (in the sense that we are given ) s.t. the following property holds. For any and , if is supported on then there exists s.t. and . is called pseudocausal when we can also guarantee that .

The underlying decision problem is then a DDT decision problem with .

Proposition: IDDT is precommitment-stable in pseudocausal precommitment games. That is, in any such game there is which is IDDT-optimal.

It should be straightforward to also formulate an analogous claim with plain DDT and iterated pseudocausal precommitment games.

To make the claim that DDT/​IDDT is precommitment-stable more often than EDT and CDT, we need to somehow compare different decision theories on the same game. For this purpose, we have the following translations.

Definition: Given an EDT precommitment game with , its DDT-translation is defined by setting

Proposition: If is formally causal then its DDT-translation is pseudocausal. Moreover, plain DDT is then precommitment-stable even without iteration.

Definition: Given a CDT precommitment game, its DDT-translation is defined by setting

Proposition: If is policy-bottlenecked then its DDT-translation is pseudocausal. Moreover, plain DDT is then precommitment-stable even without iteration.

1. ^

   Below we only use the case , in which case there is no and this simplifies to .

[Vanessa Kosoy]

The above treatment of “CDT precommitment games” is problematic: the concept made sense in the context of “FDT to CDT translation” but it’s not clear what it’s doing here (i.e. what is the first argument?) Here is a better treatment.

Definition: A CDT decision problem is the following data. We have a set of variables and for each we have its range , its set of parents and its kernel

The parent relation must induce an acyclic directed graph. We also have a selected subset of decision variables and a selected subset of outcome variables s.t. . For each there is a special element (denoting that the decision wasn’t made) and we denote . We are a given a loss function

---

This is connected to our overall formalism by setting and . We also define

The CDT counterfactuals and decision-rule are defined via a do-operator that forces to take the value unless the value was in which case it remains .

Definition: A CDT precommitment game is a CDT decision problem in which there is some special (the precommitment decision) s.t. , denoting ,
1.
2. For some we have (where denotes the decision to not precommit).
3. For every it holds that and is s.t. if has any value other than then takes the value .
4. For every , there is a special s.t. (i) (ii) (iii) is the only variable with parent (iv) is defined s.t. if has value or then has the same value as , whereas if has value then has the value .

This is connected to our abstract notion of precommitment game by setting , and for . We define to hold whenever (i) for some (ii) (iii) for every , .

The underlying decision problem of the precommitment game is constructed by deleting and identifying and in the obvious way.

The game is said to be trivial when all variables with parent are of the form or .

Proposition: CDT is precommitment-stable in trivial precommitment games.

Definition: Given a CDT precommitment game with , its DDT-translation is defined by setting and

Proposition: If is a trivial CDT precommitment game then its DDT-translation is pseudocausal. Moreover, plain DDT is precommitment stable on the translation even without iteration.