Correspondence visualizations for different interpretations of “probability”

(Writ­ten for Ar­bital in 2016.)

Re­call that there are three com­mon in­ter­pre­ta­tions of what it means to say that a coin has a 50% prob­a­bil­ity of land­ing heads:

  • The propen­sity in­ter­pre­ta­tion: Some prob­a­bil­ities are just out there in the world. It’s a brute fact about coins that they come up heads half the time; we’ll call this the coin’s phys­i­cal “propen­sity to­wards heads.” When we say the coin has a 50% prob­a­bil­ity of be­ing heads, we’re talk­ing di­rectly about this propen­sity.

  • The fre­quen­tist in­ter­pre­ta­tion: When we say the coin has a 50% prob­a­bil­ity of be­ing heads af­ter this flip, we mean that there’s a class of events similar to this coin flip, and across that class, coins come up heads about half the time. That is, the fre­quency of the coin com­ing up heads is 50% in­side the event class (which might be “all other times this par­tic­u­lar coin has been tossed” or “all times that a similar coin has been tossed” etc).

  • The sub­jec­tive in­ter­pre­ta­tion: Uncer­tainty is in the mind, not the en­vi­ron­ment. If I flip a coin and slap it against my wrist, it’s already landed ei­ther heads or tails. The fact that I don’t know whether it landed heads or tails is a fact about me, not a fact about the coin. The claim “I think this coin is heads with prob­a­bil­ity 50%” is an ex­pres­sion of my own ig­no­rance, which means that I’d bet at 1 : 1 odds (or bet­ter) that the coin came up heads.

One way to vi­su­al­ize the differ­ence be­tween these ap­proaches is by vi­su­al­iz­ing what they say about when a model of the world should count as a good model. If a per­son’s model of the world is definite, then it’s easy enough to tell whether or not their model is good or bad: We just check what it says against the facts. For ex­am­ple, if a per­son’s model of the world says “the tree is 3m tall”, then this model is cor­rect if (and only if) the tree is 3 me­ters tall.

Definite claims in the model are called “true” when they cor­re­spond to re­al­ity, and “false” when they don’t. If you want to nav­i­gate us­ing a map, you had bet­ter en­sure that the lines drawn on the map cor­re­spond to the ter­ri­tory.

But how do you draw a cor­re­spon­dence be­tween a map and a ter­ri­tory when the map is prob­a­bil­is­tic? If your model says that a bi­ased coin has a 70% chance of com­ing up heads, what’s the cor­re­spon­dence be­tween your model and re­al­ity? If the coin is ac­tu­ally heads, was the model’s claim true? 70% true? What would that mean?

The ad­vo­cate of propen­sity the­ory says that it’s just a brute fact about the world that the world con­tains on­tolog­i­cally ba­sic un­cer­tainty. A model which says the coin is 70% likely to land heads is true if and only the ac­tual phys­i­cal propen­sity of the coin is 0.7 in fa­vor of heads.

This in­ter­pre­ta­tion is use­ful when the laws of physics do say that there are mul­ti­ple differ­ent ob­ser­va­tions you may make next (with differ­ent like­li­hoods), as is some­times the case (e.g., in quan­tum physics). How­ever, when the event is de­ter­minis­tic — e.g., when it’s a coin that has been tossed and slapped down and is already ei­ther heads or tails — then this view is largely re­garded as fool­ish, and an ex­am­ple of the mind pro­jec­tion fal­lacy: The coin is just a coin, and has no spe­cial in­ter­nal struc­ture (nor spe­cial phys­i­cal sta­tus) that makes it fun­da­men­tally con­tain a lit­tle 0.7 some­where in­side it. It’s already ei­ther heads or tails, and while it may feel like the coin is fun­da­men­tally un­cer­tain, that’s a fea­ture of your brain, not a fea­ture of the coin.

How, then, should we draw a cor­re­spon­dence be­tween a prob­a­bil­is­tic map and a de­ter­minis­tic ter­ri­tory (in which the coin is already definitely ei­ther heads or tails?)

A fre­quen­tist draws a cor­re­spon­dence be­tween a sin­gle prob­a­bil­ity-state­ment in the model, and mul­ti­ple events in re­al­ity. If the map says “that coin over there is 70% likely to be heads”, and the ac­tual ter­ri­tory con­tains 10 places where 10 maps say some­thing similar, and in 7 of those 10 cases the coin is heads, then a fre­quen­tist says that the claim is true.

Thus, the fre­quen­tist pre­serves black-and-white cor­re­spon­dence: The model is ei­ther right or wrong, the 70% claim is ei­ther true or false. When the map says “That coin is 30% likely to be tails,” that (ac­cord­ing to a fre­quen­tist) means “look at all the cases similar to this case where my map says the coin is 30% likely to be tails; across all those places in the ter­ri­tory, 3/​10ths of them have a tails-coin in them.” That claim is defini­tive, given the set of “similar cases.”

By con­trast, a sub­jec­tivist gen­er­al­izes the idea of “cor­rect­ness” to al­low for shades of gray. They say, “My un­cer­tainty about the coin is a fact about me, not a fact about the coin; I don’t need to point to other ‘similar cases’ in or­der to ex­press un­cer­tainty about this case. I know that the world right in front of me is ei­ther a heads-world or a tails-world, and I have a prob­a­bil­ity dis­tri­bu­tion that puts 70% prob­a­bil­ity on heads.” They then draw a cor­re­spon­dence be­tween their prob­a­bil­ity dis­tri­bu­tion and the world in front of them, and de­clare that the more prob­a­bil­ity their model as­signs to the cor­rect an­swer, the bet­ter their model is.

If the world is a heads-world, and the prob­a­bil­is­tic map as­signed 70% prob­a­bil­ity to “heads,” then the sub­jec­tivist calls that map “70% ac­cu­rate.” If, across all cases where their map says some­thing has 70% prob­a­bil­ity, the ter­ri­tory is ac­tu­ally that way 7/​10ths of the time, then the Bayesian calls the map “well cal­ibrated”. They then seek meth­ods to make their maps more ac­cu­rate, and bet­ter cal­ibrated. They don’t see a need to in­ter­pret prob­a­bil­is­tic maps as mak­ing defini­tive claims; they’re happy to in­ter­pret them as mak­ing es­ti­ma­tions that can be graded on a slid­ing scale of ac­cu­racy.


In short, the fre­quen­tist in­ter­pre­ta­tion tries to find a way to say the model is defini­tively “true” or “false” (by iden­ti­fy­ing a col­lec­tion of similar events), whereas the sub­jec­tivist in­ter­pre­ta­tion ex­tends the no­tion of “cor­rect­ness” to al­low for shades of gray.

Fre­quen­tists some­times ob­ject to the sub­jec­tivist in­ter­pre­ta­tion, say­ing that fre­quen­tist cor­re­spon­dence is the only type that has any hope of be­ing truly ob­jec­tive. Un­der Bayesian cor­re­spon­dence, who can say whether the map should say 70% or 75%, given that the prob­a­bil­is­tic claim is not ob­jec­tively true or false ei­ther way? They claim that these sub­jec­tive as­sess­ments of “par­tial ac­cu­racy” may be in­tu­itively satis­fy­ing, but they have no place in sci­ence. Scien­tific re­ports ought to be re­stricted to fre­quen­tist state­ments, which are defini­tively ei­ther true or false, in or­der to in­crease the ob­jec­tivity of sci­ence.

Sub­jec­tivists re­ply that the fre­quen­tist ap­proach is hardly ob­jec­tive, as it de­pends en­tirely on the choice of “similar cases”. In prac­tice, peo­ple can (and do!) abuse fre­quen­tist statis­tics by choos­ing the class of similar cases that makes their re­sult look as im­pres­sive as pos­si­ble (a tech­nique known as “p-hack­ing”). Fur­ther­more, the ma­nipu­la­tion of sub­jec­tive prob­a­bil­ities is sub­ject to the iron laws of prob­a­bil­ity the­ory (which are the only way to avoid in­con­sis­ten­cies and patholo­gies when man­ag­ing your un­cer­tainty about the world), so it’s not like sub­jec­tive prob­a­bil­ities are the wild west or some­thing. Also, sci­ence has things to say about situ­a­tions even when there isn’t a huge class of ob­jec­tive fre­quen­cies we can ob­serve, and sci­ence should let us col­lect and an­a­lyze ev­i­dence even then.

For more on this de­bate, see Like­li­hood func­tions, p-val­ues, and the repli­ca­tion crisis.