The Stopped Clock Problem

When a low-prob­a­bil­ity, high-im­pact event oc­curs, and the world “got it wrong”, it is tempt­ing to look for the peo­ple who did suc­cess­fully pre­dict it in ad­vance in or­der to dis­cover their se­cret, or at least see what else they’ve pre­dicted. Un­for­tu­nately, as Wei Dai dis­cov­ered re­cently, this tends to back­fire.

It may feel a bit coun­ter­in­tu­itive, but this is ac­tu­ally fairly pre­dictable: the math backs it up on some rea­son­able as­sump­tions. First, let’s as­sume that the topic re­quired un­usual lev­els of clar­ity of thought not to be sucked into the pre­vailing (wrong) con­sen­sus: say a mere 0.001% of peo­ple ac­com­plished this. Th­ese peo­ple are worth find­ing, and listen­ing to.

But we must also note that a good chunk of the pop­u­la­tion are just pes­simists. Let’s say, very con­ser­va­tively, that 0.01% of peo­ple pre­dicted the same dis­aster just be­cause they always pre­dict the most ob­vi­ous pos­si­ble dis­aster. Sud­denly the odds are pretty good that any­body you find who suc­cess­fully pre­dicted the dis­aster is a crank. The mere fact that they cor­rectly pre­dicted the dis­aster be­comes ev­i­dence only of ex­treme rea­son­ing, but is in­suffi­cient to tell whether that rea­son­ing was ex­tremely good, or ex­tremely bad. And on bal­ance, most of the time, it’s ex­tremely bad.

Un­for­tu­nately, the prob­lem here is not just that the good pre­dic­tors are buried in a moun­tain of ran­dom oth­ers; it’s that the good pre­dic­tors are buried in a moun­tain of ex­tremely poor pre­dic­tors. The re­sult is that the mean pre­dic­tion of that group is go­ing to be no­tice­ably worse than the pre­vailing con­sen­sus on most ques­tions, not bet­ter.

Ob­vi­ously the 0.001% and 0.01% num­bers above are made up; I spent some time look­ing for real statis­tics and couldn’t find any­thing use­ful; this ar­ti­cle claims roughly 1% of Amer­i­cans are “prep­pers”, which might be a good in­di­ca­tion, ex­cept it pro­vides no source and could equally well just be the lizard­man con­stant. Re­gard­less, my point re­lies mainly on the sec­ond group be­ing an or­der of mag­ni­tude or more larger than the first, which seems (to me) fairly in­tu­itively likely to be true. If any­body has real statis­tics to prove or dis­prove this, they would be much ap­pre­ci­ated.