Absence of Evidence Is Evidence of Absence

From Robyn Dawes’s Ra­tional Choice in an Uncer­tain World:

In fact, this post-hoc fit­ting of ev­i­dence to hy­poth­e­sis was in­volved in a most grievous chap­ter in United States his­tory: the in­tern­ment of Ja­panese-Amer­i­cans at the be­gin­ning of the Se­cond World War. When Cal­ifor­nia gov­er­nor Earl War­ren tes­tified be­fore a con­gres­sional hear­ing in San Fran­cisco on Fe­bru­ary 21, 1942, a ques­tioner pointed out that there had been no sab­o­tage or any other type of es­pi­onage by the Ja­panese-Amer­i­cans up to that time. War­ren re­sponded, “I take the view that this lack [of sub­ver­sive ac­tivity] is the most om­i­nous sign in our whole situ­a­tion. It con­vinces me more than per­haps any other fac­tor that the sab­o­tage we are to get, the Fifth Column ac­tivi­ties are to get, are timed just like Pearl Har­bor was timed . . . I be­lieve we are just be­ing lul­led into a false sense of se­cu­rity.”

Con­sider War­ren’s ar­gu­ment from a Bayesian per­spec­tive. When we see ev­i­dence, hy­pothe­ses that as­signed a higher like­li­hood to that ev­i­dence gain prob­a­bil­ity, at the ex­pense of hy­pothe­ses that as­signed a lower like­li­hood to the ev­i­dence. This is a phe­nomenon of rel­a­tive like­li­hoods and rel­a­tive prob­a­bil­ities. You can as­sign a high like­li­hood to the ev­i­dence and still lose prob­a­bil­ity mass to some other hy­poth­e­sis, if that other hy­poth­e­sis as­signs a like­li­hood that is even higher.

War­ren seems to be ar­gu­ing that, given that we see no sab­o­tage, this con­firms that a Fifth Column ex­ists. You could ar­gue that a Fifth Column might de­lay its sab­o­tage. But the like­li­hood is still higher that the ab­sence of a Fifth Column would perform an ab­sence of sab­o­tage.

Let E stand for the ob­ser­va­tion of sab­o­tage, and ¬E for the ob­ser­va­tion of no sab­o­tage. The sym­bol H1 stands for the hy­poth­e­sis of a Ja­panese-Amer­i­can Fifth Column, and H2 for the hy­poth­e­sis that no Fifth Column ex­ists. The con­di­tional prob­a­bil­ity P(E | H), or “E given H,” is how con­fi­dently we’d ex­pect to see the ev­i­dence E if we as­sumed the hy­poth­e­sis H were true.

What­ever the like­li­hood that a Fifth Column would do no sab­o­tage, the prob­a­bil­ity P(¬E | H1), it won’t be as large as the like­li­hood that there’s no sab­o­tage given that there’s no Fifth Column, the prob­a­bil­ity P(¬E | H2). So ob­serv­ing a lack of sab­o­tage in­creases the prob­a­bil­ity that no Fifth Column ex­ists.

A lack of sab­o­tage doesn’t prove that no Fifth Column ex­ists. Ab­sence of proof is not proof of ab­sence. In logic, (A ⇒ B), read “A im­plies B,” is not equiv­a­lent to (¬A ⇒ ¬B), read “not-A im­plies not-B .”

But in prob­a­bil­ity the­ory, ab­sence of ev­i­dence is always ev­i­dence of ab­sence. If E is a bi­nary event and P(H | E) > P(H), i.e., see­ing E in­creases the prob­a­bil­ity of H, then P(H | ¬ E) < P(H), i.e., failure to ob­serve E de­creases the prob­a­bil­ity of H . The prob­a­bil­ity P(H) is a weighted mix of P(H | E) and P(H | ¬ E), and nec­es­sar­ily lies be­tween the two.1

Un­der the vast ma­jor­ity of real-life cir­cum­stances, a cause may not re­li­ably pro­duce signs of it­self, but the ab­sence of the cause is even less likely to pro­duce the signs. The ab­sence of an ob­ser­va­tion may be strong ev­i­dence of ab­sence or very weak ev­i­dence of ab­sence, de­pend­ing on how likely the cause is to pro­duce the ob­ser­va­tion. The ab­sence of an ob­ser­va­tion that is only weakly per­mit­ted (even if the al­ter­na­tive hy­poth­e­sis does not al­low it at all) is very weak ev­i­dence of ab­sence (though it is ev­i­dence nonethe­less). This is the fal­lacy of “gaps in the fos­sil record”—fos­sils form only rarely; it is fu­tile to trum­pet the ab­sence of a weakly per­mit­ted ob­ser­va­tion when many strong pos­i­tive ob­ser­va­tions have already been recorded. But if there are no pos­i­tive ob­ser­va­tions at all, it is time to worry; hence the Fermi Para­dox.

Your strength as a ra­tio­nal­ist is your abil­ity to be more con­fused by fic­tion than by re­al­ity; if you are equally good at ex­plain­ing any out­come you have zero knowl­edge. The strength of a model is not what it can ex­plain, but what it can’t, for only pro­hi­bi­tions con­strain an­ti­ci­pa­tion. If you don’t no­tice when your model makes the ev­i­dence un­likely, you might as well have no model, and also you might as well have no ev­i­dence; no brain and no eyes.

1 If any of this sounds at all con­fus­ing, see my dis­cus­sion of Bayesian up­dat­ing to­ward the end of The Ma­chine in the Ghost, the third vol­ume of Ra­tion­al­ity: From AI to Zom­bies.