A Bayesian Aggregation Paradox

[Jsevillamol]

In short: There is no objective way of summarizing a Bayesian update over an event with three outcomes $A:B:C$ as an update over two outcomes $A:\neg A$.

Suppose there is an event with possible outcomes $A,B,C$.
We have prior beliefs about the outcomes $p_1:p_2:p_3$.
An expert reports a likelihood factor of $e_1:e_2:e_3$.
Our posterior beliefs about $A:B:C$ are then $p_1\cdot e_1:p_2\cdot e_2:p_3\cdot e_3$.

$$\underset{\text{Prior}}{\underset{}{\left(\begin{array}{c}{p}_1\\ p_2\\ p_3\end{array}\right)}}\times \underset{\text{Update}}{\underset{}{\left(\begin{array}{c}{e}_1\\ e_2\\ e_3\end{array}\right)}}=\underset{\text{Posterior}}{\underset{}{\left(\begin{array}{c}{p}_1\cdot e_1\\ p_2\cdot e_2\\ p_3\cdot e_3\end{array}\right)}}$$

But suppose we only care about whether $A$ happens.
Our prior beliefs about $A:\neg A$ are $p_1:(p_2+p_3)$.
Our posterior beliefs are $p_1\cdot e_1:(p_2\cdot e_2+p_3\cdot e_3)$.
This implies that the likelihood factor of the expert regarding $A:\neg A$ is $\frac{p_1\cdot e_1:(p_2\cdot e_2+p_3\cdot e_3)}{p_1:(p_2+p_3)}=e_1:\frac{p_2\cdot e_2+p_3\cdot e_3}{p_2+p_3}$.

$$\underset{\text{Prior}}{\underset{}{\left(\begin{array}{c}{p}_1\\ p_2+p_3\end{array}\right)}}\times \underset{\text{Update}}{\underset{}{\left(\begin{array}{c}{e}_1\\ \frac{p_2\cdot e_2+p_3\cdot e_3}{p_2+p_3}\end{array}\right)}}=\underset{\text{Posterior}}{\underset{}{\left(\begin{array}{c}{p}_1\cdot e_1\\ p_2\cdot e_2+p_3\cdot e_3\end{array}\right)}}$$

This likelihood factor depends on the ratio of prior beliefs $p_2:p_3$.

Concretely, the lower factor in the update is the weighted mean of the evidence $e_2$ and $e_3$ according to the weights $p_2$ and $p_3$.

This has a relatively straightforward interpretation. The update is supposed to be the ratio of the likelihoods under each hypothesis. The upper factor in the update is $P\left(E|A\right)$. The lower factor is $P(E|B\cup C)=\frac{P\left(B\right)\cdot P\left(E|B\right)+P\left(C\right)\cdot P\left(E|C\right)}{P\left(B\right)+P\left(C\right)}$.

$$\underset{\text{Posterior}}{\underset{}{\left(\begin{array}{c}P\left(A|E\right)\\ P(B\cup C|E)\end{array}\right)}}\propto \underset{\text{Prior}}{\underset{}{\left(\begin{array}{c}P\left(A\right)\\ P(B\cup C)\end{array}\right)}}\times \underset{\text{Update}}{\underset{}{\left(\begin{array}{c}P\left(E|A\right)\\ P(E|B\cup C)\end{array}\right)}}$$$$\underset{\text{Update}}{\underset{}{\left(\begin{array}{c}P\left(E|A\right)\\ P(E|B\cup C)\end{array}\right)}}=\left(\begin{array}{c}P\left(E|A\right)\\ \frac{P(E\cap (B\cup C\left)\right)}{P(B\cup C)}\end{array}\right)=\left(\begin{array}{c}P\left(E|A\right)\\ \frac{P\left(B\right)\cdot P\left(E|B\right)+P\left(C\right)\cdot P\left(E|C\right)}{P\left(B\right)+P\left(C\right)}\end{array}\right)$$

I found this very surprising—the summary of the expert report depends on my prior beliefs!

I claim that this phenomena is unintuitive, and being unaware of this can lead to errors.

Why this is weird

Bayes’ rule describes how to update our prior beliefs using data.

In my mind, one very nice property of Bayes rule was that it cleanly separates the process into a subjective part (eliciting your priors) and an ~objective part (computing the update).

$$\text{Posterior}=\underset{\text{Subjective}}{\underset{}{\text{Prior}}}\times \underset{\text{Objective}}{\underset{}{\text{Likelihood}}}$$

For example, we may disagree on our prior beliefs on whether eg COVID19 originated in a lab. But we cannot disagree on the direction and magnitude of the update caused by learning that it originated in one of the few cities in the world with a gain-of-function lab working on coronaviruses.

Because of this, researchers are encouraged to report their update factors together with their all considered beliefs. This way, users can use their research for their own conclusions by multiplying their prior with the update. And metastudies can just take the product of the likelihoods of all studies to estimate the combined effect of the evidence.

In the above example, we lose this nice property—the update factor depends on the prior beliefs of the user. Researchers would not be able to objectively summarize their likelihood about whether COVID19 originated in a lab accidentally vs zoonotically vs being designed as a bioweapon as a single number for people who only care about whether it originated in a lab versus any other possibility.

Examples in the wild

I ran into this problem twice recently:

1. When analyzing Mennen’s ABC example of a case where averaging the logarithmic odds of experts seems to result in nonsense.

2. In my own research on interpreting Bayesian Networks as I was trying to come up with a way of decomposing a Bayesian update into a combination of several updates.

In both cases being unaware of the phenomena led me to a conceptual mistake.

Mennen’s ABC example

Mennen’s example involves three experts debating an event with three possible outcomes, $A:B:C$.

Expert #1 assigns relative odds of $2:1:1$.
Expert #2 assigns relative odds of $1:2:1$.
Expert #3 assigns relative odds of $1:1:2$.

The logodds-averaging pooled opinion of the experts is $\sqrt[3]{32}:\sqrt[3]{32}:\sqrt[3]{32}$ i.e. equal odds, which correspond to a probability of $A$ equal to $\frac{1}{3}\approx 33.33\%$.

$$\sqrt[3]{3\underset{\text{Expert \#1}}{\underset{}{\left(\begin{array}{c}2\\ 1\\ 1\end{array}\right)}}\times \underset{\text{Expert \#2}}{\underset{}{\left(\begin{array}{c}1\\ 2\\ 1\end{array}\right)}}\times \underset{\text{Expert \#3}}{\underset{}{\left(\begin{array}{c}1\\ 1\\ 2\end{array}\right)}}}=\underset{\text{Pooled opinion}}{\underset{}{\left(\begin{array}{c}\sqrt[3]{32}\\ \sqrt[3]{32}\\ \sqrt[3]{32}\end{array}\right)}}$$

But suppose we only care about $A:\neg A$.

Expert #1’s implicit odds are $2:2$.
Expert #2’s implicit odds are $1:3$.
Expert #3’s implicit odds are $1:3$.

The pooled odds in this case are $\sqrt[3]{32}:\sqrt[3]{32\cdot 3\cdot 3}$, which correspond to a probability of $A$ equal to $\frac{\sqrt[3]{32}}{\sqrt[3]{32}+\sqrt[3]{32\cdot 3\cdot 3}}\approx 32.47\%$.

$$\sqrt[3]{3\underset{\text{Expert \#1}}{\underset{}{\left(\begin{array}{c}2\\ 1+1\end{array}\right)}}\times \underset{\text{Expert \#2}}{\underset{}{\left(\begin{array}{c}1\\ 2+1\end{array}\right)}}\times \underset{\text{Expert \#3}}{\underset{}{\left(\begin{array}{c}1\\ 1+2\end{array}\right)}}}=\underset{\text{Pooled opinion}}{\underset{}{\left(\begin{array}{c}\sqrt[3]{32}\\ \sqrt[3]{32\times 3\times 3}\end{array}\right)}}$$

We get different results depending on whether we take the implicit odds after or before pooling expert opinion. What is going on?

Mennen claims that this is a strike against logarithmic pooling. The issue according to him is in the step where we take the opinion of the three experts and aggregate it using average logodds.

I think that this is related to the phenomena I described at the beginning of the article. The problem is with the step where we take the relative odds $1:2:1$ and summarize them as $1:3$.

It’s no wonder that logodd pooling gives inconsistent results when we aggregate outcomes. Bayesian updating is not well defined in that case!

Interpreting Bayesian Networks

I will not enter into too much detail because my theory of interpretability of Bayesian Networks is very complex. But it suffices to say that I was getting inconsistent results because of this issue.

In essence, I came up with a way of decomposing a Bayesian update into a series of independent steps, corresponding to different subgraphs of a Bayesian Network.

For example, I would decompose the update over a node with three outcomes $A,B,C$ as the product of the baseline odds of the event and a number of updates.

In my system, I only cared about whether $A$ happened. So I naively summarized each update before aggregating them.

$$O\left(\text{Event}|\text{Evidence}\right)\approx \underset{\text{Prior}}{\underset{}{\left(\begin{array}{c}{p}_1\\ p_2+p_3\end{array}\right)}}\times \underset{\text{Argument 1}}{\underset{}{\left(\begin{array}{c}{e}_{1,1}\\ e_{1,2}+e_{1,3}\end{array}\right)}}\times \cdots \times \underset{\text{Argument n}}{\underset{}{\left(\begin{array}{c}{e}_{n,1}\\ e_{n,2}+e_{n,3}\end{array}\right)}}$$

This was giving me very poor results—my resulting updates would be very off compared to traditional inference algorithms like message passing.

It is no wonder this was giving me bad results—it is the wrong way of going about it! Our analysis at the beginning implies that the update should be the average of $e_{i,2}$ and $e_{i,3}$, instead of the sum.

After realizing the paradox, I changed my system to not summarizing the odds of $A:\neg A$ until after aggregating all the updates.

$$O\left(\text{Event}|\text{Evidence}\right)\approx \underset{\text{Prior}}{\underset{}{\left(\begin{array}{c}{p}_1\\ p_2\\ p_3\end{array}\right)}}\times \underset{\text{Argument 1}}{\underset{}{\left(\begin{array}{c}{e}_{1,1}\\ e_{1,2}\\ e_{1,3}\end{array}\right)}}\times \cdots \times \underset{\text{Argument n}}{\underset{}{\left(\begin{array}{c}{e}_{n,1}\\ e_{n,2}\\ e_{n,3}\end{array}\right)}}$$

Performance improved.

Consequences

I am quite confused about what to think about this.

It clearly has consequences, as illustrated by the examples in the previous section. But I am not sure what to recommend doing in response.

My most immediate takeaway is to be very careful when aggregating outcomes—there is an important chance we will be introducing an error along the way.

Beyond that, the aggregation paradox seems to imply that we need to work at the correct level of aggregation. We cannot naively deduce implied binary odds from the distribution of a multiple outcome event.

But what is the right level of aggregation?

When aggregating, the lower factor of the update is a weighted mean of the evidence likelihoods $P\left(E|B\right)$ and $P\left(E|C\right)$. This suggests that the problem disappears when we impose $P\left(E|B\right)=P\left(E|C\right)$ for any disaggregation of the joint event $\neg A$ into subevents $B$ and $C$.

But this condition is too strong. For example, we could base our disaggregation on the observed evidence. For example, if the evidence $E$ can either be $\text{Red}$ or $\text{Blue}$ we could disaggregate ~A into the cases where $E=\text{Red}$ and the cases where $E=\text{Blue}$. In that case, the condition cannot ever be satisfied, by definition.

We can say that this disaggregation is not a sensible one, and ought to be excluded for the purposes of the condition. But in that case we have passed the bucket down to defining what is a sensible disaggregation.

Another approach is to assume that the prior relative likelihood of any aggregated outcomes is uniform, ie $P\left(B\right)=P\left(C\right)$. In that case, we have that $P(E|B\cup C)=\frac{P\left(B\right)\cdot P\left(E|B\right)+P\left(C\right)\cdot P\left(E|C\right)}{P\left(B\right)+P\left(C\right)}=\frac{P\left(E|B\right)+P\left(E|C\right)}{2}$.

But then we can no longer chain updates—after applying any likelihood where $P\left(E|B\right)\ne P\left(E|C\right)$ the resulting posterior will no longer meet this condition.

Pragmatically, it seems like the best we can do if we want to rescue objetivity is to resign ourselfs to summarize the updates assuming a uniform prior. That is, by averaging the evidence associated to each aggregated outcome.

This is not enough to correctly approximate Bayesian updating, as we can see in the example below:

$$\underset{\text{Posterior}}{\underset{}{\left(\begin{array}{c}1\\ 0.01\\ 0.01\end{array}\right)}}=\underset{\text{Prior}}{\underset{}{\left(\begin{array}{c}1\\ 1\\ 1\end{array}\right)}}\times \underset{\text{Refute B}}{\underset{}{\left(\begin{array}{c}1\\ 0.01\\ 1\end{array}\right)}}\times \underset{\text{Refute C}}{\underset{}{\left(\begin{array}{c}1\\ 1\\ 0.01\end{array}\right)}}\ne \underset{\text{Prior}}{\underset{}{\left(\begin{array}{c}1\\ 1+1\end{array}\right)}}\times \underset{\text{Refute B}}{\underset{}{\left(\begin{array}{c}1\\ \frac{0.01+1}{2}\end{array}\right)}}\times \underset{\text{Refute C}}{\underset{}{\left(\begin{array}{c}1\\ \frac{1+0.01}{2}\end{array}\right)}}\approx \underset{\text{Posterior}}{\underset{}{\left(\begin{array}{c}1\\ 0.5\end{array}\right)}}$$

But I can’t see how to do better in the absence of more information.

One key takeaway here is that beliefs and updates are summarized in different ways.

$$\underset{\text{Belief}}{\underset{}{\left(\begin{array}{c}{p}_1\\ p_2\\ p_3\end{array}\right)}}\to \underset{\text{Summarized belief}}{\underset{}{\left(\begin{array}{c}{p}_1\\ p_2+p_3\end{array}\right)}}\underset{\text{Update}}{\underset{}{\left(\begin{array}{c}{e}_1\\ e_2\\ e_3\end{array}\right)}}\to \underset{\text{Summarized update}}{\underset{}{\left(\begin{array}{c}{e}_1\\ \frac{e_2+e_3}{2}\end{array}\right)}}$$

In summary

I have explained one counterintuitive consequence of Bayesian updating on variables with more than two outcomes. This paradox implies that we should be careful when grouping together outcomes of a variable. And I have shown two situations where this unintuitive consequence is relevant.

This is a post meant to explore and start a discussion more than provide definite answers. Some things I’d be keen on discussing include:

1. Is this a documented phenomena? Where can I find more discussion?

2. What does this imply for formulating forecasting questions? Will this result in problems when asking binary questions about events that are multifaceted?

3. What is “the right level” of outcome aggregation for a given problem?

4. Are there other examples where similar issues come up?

I’d be really interested in your thoughts—please leave a comment if you have any!

Acknowledgements

Thanks to rossry, Nuño Sempere, Eric Neyman, Ehud Reiter and ForgedInvariant for discussing this topic with me and helping me clarify some ideas.

Thanks to Alex Mennen for coming up with the example I referenced in the post.