How Much Evidence Does It Take?

Pre­vi­ously, I defined ev­i­dence as “an event en­tan­gled, by links of cause and effect, with what­ever you want to know about,” and en­tan­gled as “hap­pen­ing differ­ently for differ­ent pos­si­ble states of the tar­get.” So how much en­tan­gle­ment—how much ra­tio­nal ev­i­dence—is re­quired to sup­port a be­lief?

Let’s start with a ques­tion sim­ple enough to be math­e­mat­i­cal: How hard would you have to en­tan­gle your­self with the lot­tery in or­der to win? Sup­pose there are sev­enty balls, drawn with­out re­place­ment, and six num­bers to match for the win. Then there are 131,115,985 pos­si­ble win­ning com­bi­na­tions, hence a ran­domly se­lected ticket would have a 1131,115,985 prob­a­bil­ity of win­ning (0.0000007%). To win the lot­tery, you would need ev­i­dence se­lec­tive enough to visi­bly fa­vor one com­bi­na­tion over 131,115,984 al­ter­na­tives.

Sup­pose there are some tests you can perform which dis­crim­i­nate, prob­a­bil­is­ti­cally, be­tween win­ning and los­ing lot­tery num­bers. For ex­am­ple, you can punch a com­bi­na­tion into a lit­tle black box that always beeps if the com­bi­na­tion is the win­ner, and has only a 14 (25%) chance of beep­ing if the com­bi­na­tion is wrong. In Bayesian terms, we would say the like­li­hood ra­tio is 4 to 1. This means that the box is 4 times as likely to beep when we punch in a cor­rect com­bi­na­tion, com­pared to how likely it is to beep for an in­cor­rect com­bi­na­tion.

There are still a whole lot of pos­si­ble com­bi­na­tions. If you punch in 20 in­cor­rect com­bi­na­tions, the box will beep on 5 of them by sheer chance (on av­er­age). If you punch in all 131,115,985 pos­si­ble com­bi­na­tions, then while the box is cer­tain to beep for the one win­ning com­bi­na­tion, it will also beep for 32,778,996 los­ing com­bi­na­tions (on av­er­age).

So this box doesn’t let you win the lot­tery, but it’s bet­ter than noth­ing. If you used the box, your odds of win­ning would go from 1 in 131,115,985 to 1 in 32,778,997. You’ve made some progress to­ward find­ing your tar­get, the truth, within the huge space of pos­si­bil­ities.

Sup­pose you can use an­other black box to test com­bi­na­tions twice, in­de­pen­dently. Both boxes are cer­tain to beep for the win­ning ticket. But the chance of a box beep­ing for a los­ing com­bi­na­tion is 14 in­de­pen­dently for each box; hence the chance of both boxes beep­ing for a los­ing com­bi­na­tion is 116. We can say that the cu­mu­la­tive ev­i­dence, of two in­de­pen­dent tests, has a like­li­hood ra­tio of 16:1. The num­ber of los­ing lot­tery tick­ets that pass both tests will be (on av­er­age) 8,194,749.

Since there are 131,115,985 pos­si­ble lot­tery tick­ets, you might guess that you need ev­i­dence whose strength is around 131,115,985 to 1—an event, or se­ries of events, which is 131,115,985 times more likely to hap­pen for a win­ning com­bi­na­tion than a los­ing com­bi­na­tion. Ac­tu­ally, this amount of ev­i­dence would only be enough to give you an even chance of win­ning the lot­tery. Why? Be­cause if you ap­ply a filter of that power to 131 mil­lion los­ing tick­ets, there will be, on av­er­age, one los­ing ticket that passes the filter. The win­ning ticket will also pass the filter. So you’ll be left with two tick­ets that passed the filter, only one of them a win­ner. Fifty per­cent odds of win­ning, if you can only buy one ticket.

A bet­ter way of view­ing the prob­lem: In the be­gin­ning, there is 1 win­ning ticket and 131,115,984 los­ing tick­ets, so your odds of win­ning are 1:131,115,984. If you use a sin­gle box, the odds of it beep­ing are 1 for a win­ning ticket and 0.25 for a los­ing ticket. So we mul­ti­ply 1:131,115,984 by 1:0.25 and get 1:32,778,996. Ad­ding an­other box of ev­i­dence mul­ti­plies the odds by 1:0.25 again, so now the odds are 1 win­ning ticket to 8,194,749 los­ing tick­ets.

It is con­ve­nient to mea­sure ev­i­dence in bits—not like bits on a hard drive, but math­e­mat­i­cian’s bits, which are con­cep­tu­ally differ­ent. Math­e­mat­i­cian’s bits are the log­a­r­ithms, base 12, of prob­a­bil­ities. For ex­am­ple, if there are four pos­si­ble out­comes A, B, C, and D, whose prob­a­bil­ities are 50%, 25%, 12.5%, and 12.5%, and I tell you the out­come was “D,” then I have trans­mit­ted three bits of in­for­ma­tion to you, be­cause I in­formed you of an out­come whose prob­a­bil­ity was 18.

It so hap­pens that 131,115,984 is slightly less than 2 to the 27th power. So 14 boxes or 28 bits of ev­i­dence—an event 268,435,456:1 times more likely to hap­pen if the ticket-hy­poth­e­sis is true than if it is false—would shift the odds from 1:131,115,984 to 268,435,456:131,115,984, which re­duces to 2:1. Odds of 2 to 1 mean two chances to win for each chance to lose, so the prob­a­bil­ity of win­ning with 28 bits of ev­i­dence is 23. Ad­ding an­other box, an­other 2 bits of ev­i­dence, would take the odds to 8:1. Ad­ding yet an­other two boxes would take the chance of win­ning to 128:1.

So if you want to li­cense a strong be­lief that you will win the lot­tery—ar­bi­trar­ily defined as less than a 1% prob­a­bil­ity of be­ing wrong—34 bits of ev­i­dence about the win­ning com­bi­na­tion should do the trick.

In gen­eral, the rules for weigh­ing “how much ev­i­dence it takes” fol­low a similar pat­tern: The larger the space of pos­si­bil­ities in which the hy­poth­e­sis lies, or the more un­likely the hy­poth­e­sis seems a pri­ori com­pared to its neigh­bors, or the more con­fi­dent you wish to be, the more ev­i­dence you need.

You can­not defy the rules; you can­not form ac­cu­rate be­liefs based on in­ad­e­quate ev­i­dence. Let’s say you’ve got 10 boxes lined up in a row, and you start punch­ing com­bi­na­tions into the boxes. You can­not stop on the first com­bi­na­tion that gets beeps from all 10 boxes, say­ing, “But the odds of that hap­pen­ing for a los­ing com­bi­na­tion are a mil­lion to one! I’ll just ig­nore those ivory-tower Bayesian rules and stop here.” On av­er­age, 131 los­ing tick­ets will pass such a test for ev­ery win­ner. Con­sid­er­ing the space of pos­si­bil­ities and the prior im­prob­a­bil­ity, you jumped to a too-strong con­clu­sion based on in­suffi­cient ev­i­dence. That’s not a pointless bu­reau­cratic reg­u­la­tion; it’s math.

Of course, you can still be­lieve based on in­ad­e­quate ev­i­dence, if that is your whim; but you will not be able to be­lieve ac­cu­rately. It is like try­ing to drive your car with­out any fuel, be­cause you don’t be­lieve in the fuddy-duddy con­cept that it ought to take fuel to go places. Wouldn’t it be so much more fun, and so much less ex­pen­sive, if we just de­cided to re­peal the law that cars need fuel?

Well, you can try. You can even shut your eyes and pre­tend the car is mov­ing. But re­ally ar­riv­ing at ac­cu­rate be­liefs re­quires ev­i­dence-fuel, and the fur­ther you want to go, the more fuel you need.