Prediction Markets Don’t Reveal The Territory

[A draft sec­tion from a longer piece I am writ­ing on pre­dic­tion and fore­cast­ing. Epistemic Sta­tus: I don’t know what I am miss­ing, and I am filled with doubt and un­cer­tainty.]

If the no­tion of pro­fes­sional fore­cast­ers dis­turbs you in your sleep, and you toss and turn wor­ry­ing about the blight of ex­perts brood­ing upon the world, per­haps the golden light of dis­tributed in­for­ma­tion sys­tems have peaked out from be­yond these dark­est vi­sions, and you have hope for the wis­dom of crowds.

Pre­dic­tion mar­kets ag­gre­gate in­for­ma­tion by in­cen­tiviz­ing pre­dic­tors to place bets on the out­comes of well-defined ques­tions. Since in­for­ma­tion can be both niche and use­ful, pre­dic­tion mar­kets also in­cen­tivize the de­vel­op­ment of spe­cial­ized ex­per­tise that is then in­cor­po­rated into the gen­eral pool of in­for­ma­tion in the form of a bet. When this works, it works very well.

When in­for­ma­tion is not widely dis­tributed or dis­cov­er­able, pre­dic­tion mar­kets are not use­ful. Pre­dic­tion mar­kets for WMDs, Pope Fran­cis’ next pro­nounce­ment, or which celebrity cou­ple will choose to live in a van prob­a­bly will not work. Or con­sider a pub­lic pre­dic­tion mar­ket about what per­cent of the cur­rent fresh­man class at Cal­ifor­nia pub­lic uni­ver­si­ties will make it to grad­u­a­tion. Such a mar­ket would be pretty dis­torted if all the reg­is­trars and ad­mis­sions coun­cillors were bet­ting as well.

Pre­dic­tion mar­kets do have some wicked clever uses too. For ex­am­ple, a pre­dic­tion mar­ket can also act as a proxy for some other event. That is to say, that through some in­ge­nious de­sign one can cor­re­late a pre­dic­tion mar­ket’s as­sess­ment of an event to an­other mea­surable out­come. Here is one in­stance in which re­searchers used a pre­dic­tion mar­ket about the prob­a­bil­ity of war in Iraq, cor­re­lated it to the price of oil, and es­ti­mated the effect of war on oil prices. This pro­vided the very use­ful in­for­ma­tion tel­ling us what % of the price of oil is caused by the threat of war in Iraq. At an even broader level, this pre­dic­tion mar­ket de­sign al­lows us to study the effects of war on economies.

On the other hand, an ad­di­tional limi­ta­tion to pre­dic­tion mar­kets is that peo­ple have to be in­ter­ested enough to take part in them, which is a real bum­mer. In­tel­li­gent quan­ti­ta­tive peo­ple might en­joy re­search­ing to gain some bet­ting lev­er­age in pre­dic­tion mar­kets qua pre­dic­tion mar­kets. But even then, most peo­ple want to re­search ques­tions that they them­selves find in­ter­est­ing [cita­tion needed]. So even the best de­signed pre­dic­tion mar­ket can fail with­out enough par­ties in­cen­tivized to care.

The great­est limi­ta­tion for pre­dic­tion mar­kets how­ever is not any of the above tech­ni­cal prob­lems. We are op­ti­mistic that these can be over­come. But there is a log­i­cal prob­lem which can’t. Since each spe­cial­ized piece of in­for­ma­tion is con­verted into a bet, the mar­ket will re­act to that new in­for­ma­tion with­out hav­ing to know the par­tic­u­lars of that in­for­ma­tion. This is the beau­tiful won­der of mar­kets—ev­ery­thing is turned into a util­ity func­tion. How­ever, for boards, ad­minis­tra­tors, and gov­ern­ments which want to take ac­tion based upon the in­for­ma­tion from a pre­dic­tion mar­ket two bits of im­por­tant in­for­ma­tion are left to­tally in­ac­cessible. First, what in­for­ma­tion was the most salient for bring­ing the mar­ket odds where they cur­rently are? Se­condly, what as­pect of the cur­rent state of af­fairs is the most lev­er­age­able? That is, of all the hid­den fac­tors which caused the mar­ket con­sen­sus to reach this point, which, if any of them, do we have any power to af­fect? If the point is to not just know what the mar­ket says, but to know how the world works, then pre­dic­tion mar­kets in them­selves may not be of much help. Here are two quick ex­am­ples to demon­strate illus­trate the point:

You work at the North Pole man­ag­ing pre­sent-pro­cure­ment for heads of state (PP-HOS, for short). Each year you scram­ble to get enough coal for the var­i­ous heads of state be­cause you don’t know un­til Christ­mas week whether they are on the naughty or nice list. This is an im­por­tant ques­tion be­cause heads of state re­ceive coal pro­por­tional to their stand­ing in so­ciety, and since the cost of coal rises in win­ter, it costs your de­part­ment quite a bit of money to ex­pe­d­ite all these coal or­ders. So this year you have cre­ated a pre­dic­tion mar­ket to tell you the chances of the pres­i­dent of Hun­gary get­ting coal again this year and you plan on act­ing on the mar­ket’s pre­dic­tion in Septem­ber, well ahead of the Novem­ber coal rush…. The mar­ket is a suc­cess! Your de­part­ment saves some money, and you save just about the right amount of coal for the be­loved pres­i­dent of Hun­gary. But when the big man pulls the plug on fund­ing the mar­ket ap­para­tus, you re­al­ize that de­spite all the lit­tle helpers that made the mar­ket a suc­cess, you didn’t gain any wis­dom about how to pre­dict whether a head of state will get coal this year from it. That is an ex­am­ple of a mar­ket work­ing with­out con­vey­ing any in­sights. Thus mar­kets keep in­scrutable the in­ner work­ings of Father Christ­mas’ naughty list.

The sec­ond ex­am­ple demon­strates the lev­er­age prob­lem of a mar­ket. You are the prin­ci­pal of a school. You get drunk one night and rent out a Ve­gas cas­ino which you re­vamp into a test score bet­ting com­plex. You want to know how your stu­dents will do on this week’s stan­dard­ized test. So you make all their in­for­ma­tion available to pa­trons who then place bets on each stu­dent. De­spite the drugs, sex, and al­co­hol in this ad­minis­tra­tive Bac­cha­nal, the mar­ket works as­tound­ingly well, and the pre­dicted in­di­vi­d­ual perfor­mance on the stan­dard­ized tests matches the ac­tual perfor­mance near perfectly. How­ever, in the sober light of late af­ter­noon, you re­al­ize that your mar­ket solu­tion for pre­dict­ing scores didn’t re­veal much about what you should be do­ing differ­ently. In fact, the post-mortem in­di­cates that the 3 biggest pre­dic­tors of test scores are not things even re­motely un­der your con­trol. You de­spair be­liev­ing that there is noth­ing you can do to help stu­dents im­prove. Even if there were a fourth cause of test suc­cess which is un­der your con­trol, it doesn’t mat­ter and will not be dis­cernible among the thou­sands of bets made, be­cause it, like ev­ery­thing else was flat­tened into the same util­ity func­tion.