Prediction Markets Don’t Reveal The Territory

[A draft section from a longer piece I am writing on prediction and forecasting. Epistemic Status: I don’t know what I am missing, and I am filled with doubt and uncertainty.]

If the notion of professional forecasters disturbs you in your sleep, and you toss and turn worrying about the blight of experts brooding upon the world, perhaps the golden light of distributed information systems have peaked out from beyond these darkest visions, and you have hope for the wisdom of crowds.

Prediction markets aggregate information by incentivizing predictors to place bets on the outcomes of well-defined questions. Since information can be both niche and useful, prediction markets also incentivize the development of specialized expertise that is then incorporated into the general pool of information in the form of a bet. When this works, it works very well.

When information is not widely distributed or discoverable, prediction markets are not useful. Prediction markets for WMDs, Pope Francis’ next pronouncement, or which celebrity couple will choose to live in a van probably will not work. Or consider a public prediction market about what percent of the current freshman class at California public universities will make it to graduation. Such a market would be pretty distorted if all the registrars and admissions councillors were betting as well.

Prediction markets do have some wicked clever uses too. For example, a prediction market can also act as a proxy for some other event. That is to say, that through some ingenious design one can correlate a prediction market’s assessment of an event to another measurable outcome. Here is one instance in which researchers used a prediction market about the probability of war in Iraq, correlated it to the price of oil, and estimated the effect of war on oil prices. This provided the very useful information telling us what % of the price of oil is caused by the threat of war in Iraq. At an even broader level, this prediction market design allows us to study the effects of war on economies.

On the other hand, an additional limitation to prediction markets is that people have to be interested enough to take part in them, which is a real bummer. Intelligent quantitative people might enjoy researching to gain some betting leverage in prediction markets qua prediction markets. But even then, most people want to research questions that they themselves find interesting [citation needed]. So even the best designed prediction market can fail without enough parties incentivized to care.

The greatest limitation for prediction markets however is not any of the above technical problems. We are optimistic that these can be overcome. But there is a logical problem which can’t. Since each specialized piece of information is converted into a bet, the market will react to that new information without having to know the particulars of that information. This is the beautiful wonder of markets—everything is turned into a utility function. However, for boards, administrators, and governments which want to take action based upon the information from a prediction market two bits of important information are left totally inaccessible. First, what information was the most salient for bringing the market odds where they currently are? Secondly, what aspect of the current state of affairs is the most leverageable? That is, of all the hidden factors which caused the market consensus to reach this point, which, if any of them, do we have any power to affect? If the point is to not just know what the market says, but to know how the world works, then prediction markets in themselves may not be of much help. Here are two quick examples to demonstrate illustrate the point:

You work at the North Pole managing present-procurement for heads of state (PP-HOS, for short). Each year you scramble to get enough coal for the various heads of state because you don’t know until Christmas week whether they are on the naughty or nice list. This is an important question because heads of state receive coal proportional to their standing in society, and since the cost of coal rises in winter, it costs your department quite a bit of money to expedite all these coal orders. So this year you have created a prediction market to tell you the chances of the president of Hungary getting coal again this year and you plan on acting on the market’s prediction in September, well ahead of the November coal rush…. The market is a success! Your department saves some money, and you save just about the right amount of coal for the beloved president of Hungary. But when the big man pulls the plug on funding the market apparatus, you realize that despite all the little helpers that made the market a success, you didn’t gain any wisdom about how to predict whether a head of state will get coal this year from it. That is an example of a market working without conveying any insights. Thus markets keep inscrutable the inner workings of Father Christmas’ naughty list.

The second example demonstrates the leverage problem of a market. You are the principal of a school. You get drunk one night and rent out a Vegas casino which you revamp into a test score betting complex. You want to know how your students will do on this week’s standardized test. So you make all their information available to patrons who then place bets on each student. Despite the drugs, sex, and alcohol in this administrative Bacchanal, the market works astoundingly well, and the predicted individual performance on the standardized tests matches the actual performance near perfectly. However, in the sober light of late afternoon, you realize that your market solution for predicting scores didn’t reveal much about what you should be doing differently. In fact, the post-mortem indicates that the 3 biggest predictors of test scores are not things even remotely under your control. You despair believing that there is nothing you can do to help students improve. Even if there were a fourth cause of test success which is under your control, it doesn’t matter and will not be discernible among the thousands of bets made, because it, like everything else was flattened into the same utility function.