Value of Information: Four Examples

Value of In­for­ma­tion (VoI) is a con­cept from de­ci­sion anal­y­sis: how much an­swer­ing a ques­tion al­lows a de­ci­sion-maker to im­prove its de­ci­sion. Like op­por­tu­nity cost, it’s easy to define but of­ten hard to in­ter­nal­ize; and so in­stead of be­la­bor­ing the defi­ni­tion let’s look at some ex­am­ples.

Gam­bling with Bi­ased Coins

Nor­mal coins are ap­prox­i­mately fair.1 Sup­pose you and your friend want to gam­ble, and fair coins are bor­ing, so he takes out a quar­ter and some gum and sticks the gum to the face of the quar­ter near the edge. He then offers to pay you $24 if the coin lands gum down, so long as you pay him $12 to play the game. Should you take that bet?

First, let’s as­sume risk neu­tral­ity for the amount of money you’re wa­ger­ing. Your ex­pected profit is $24p-12, where p is the prob­a­bil­ity the coin lands gum down. This is a good deal if p>.5, but a bad deal if p<.5. So… what’s p? More im­por­tantly, how much should you pay to figure out p?

A Bayesian rea­soner look­ing at this prob­lem first tries to put a prior on p. An easy choice is a uniform dis­tri­bu­tion be­tween 0 and 1, but there are a lot of rea­sons to be un­com­fortable with that dis­tri­bu­tion. It might be that the gum will be more likely to be on the bot­tom- but it also might be more likely to be on the top. The gum might not skew the re­sults very much- or it might skew them mas­sively. You could choose a differ­ent prior, but you’d have trou­ble jus­tify­ing it be­cause you don’t have any solid ev­i­dence to up­date on yet.2

If you had a uniform prior and no ad­di­tional ev­i­dence, then the deal as offered is neu­tral. But be­fore you choose to ac­cept or re­ject, your friend offers you an­other deal- he’ll flip the coin once and let you see the re­sult be­fore you choose to take the $12 deal, but you can’t win any­thing on this first flip. How much should you pay to see one flip?

Start by mod­el­ing your­self af­ter you see one flip. It’ll ei­ther come up gum or no gum, and you’ll up­date and pro­duce a pos­te­rior for each case. In the first case, your pos­te­rior on p is P(p)=2p; in the sec­ond, P(p)=2-2p. Your ex­pected profit for play­ing in the first case is $4;3 your ex­pected profit for play­ing in the sec­ond case is nega­tive $4. You think there’s a half chance it’ll land gum side up, and a half chance it’ll land gum side down, and if it lands gum side down you can choose not to play. There’s a half chance you get $4 from see­ing the flip, and a half chance you get noth­ing (be­cause you don’t play) from see­ing the flip, and so $2 is the VoI of see­ing one flip of the bi­ased coin, given your origi­nal prior.

No­tice that, even though it’d be im­pos­si­ble to figure out the ‘true’ chance that the coin will land gum down, you can model how much it would be worth it to you to figure that out. If I were able to tell you p di­rectly, then you could choose to gam­ble only when p>.5, and you would earn an av­er­age of $3.4 One coin flip gives you two thirds of the value that perfect in­for­ma­tion would give you.

Also no­tice that you need to change your de­ci­sion to get any value out of more in­for­ma­tion. Sup­pose that, in­stead of let­ting you choose whether or not to gam­ble, your friend made you de­cide, flipped two coins, and then paid you if the sec­ond coin landed gum down and you paid him. The coin is flipped the same num­ber of times, but you’re worse off be­cause you have to de­cide with less in­for­ma­tion.

It’s also worth not­ing that mul­ti­modal dis­tri­bu­tions- where there are strong clusters rather than smooth land­scapes- tend to have higher VoI. If we knew the bi­ased coin would ei­ther always come up heads or always come up tails, and ex­pected each case were equally likely, then see­ing one flip is worth $6, be­cause it’s a half chance of a guaran­teed $12.

Choos­ing where to invest

Here’s an ex­am­ple I came across in my re­search:

Klein­muntz and Willis were try­ing to de­ter­mine the value of do­ing de­tailed anti-ter­ror­ism as­sess­ments in the state of Cal­ifor­nia for the Depart­ment of Home­land Se­cu­rity. There are hun­dreds of crit­i­cal in­fras­truc­ture sites across the state, and it’s sim­ply not pos­si­ble to do a de­tailed anal­y­sis of each site. There are ter­ror­ism ex­perts, though, who can quickly provide an es­ti­mate of the risk to var­i­ous sites.

They gave a care­fully de­signed sur­vey to those ex­perts, ask­ing them to rate the rel­a­tive prob­a­bil­ity that a site would be at­tacked (con­di­tioned on an at­tack oc­cur­ring) and the prob­a­bil­ity that an at­tack would suc­ceed on a scale from 0 to 10, and the scale of fatal­ities and eco­nomic loss on a log­a­r­ith­mic scale from 0 to 7. The ex­perts were com­fortable with the sur­vey5 and able to give mean­ingful an­swers.

Now Klein­mutz and Willis were able to take the elic­ited vuln­er­a­bil­ity es­ti­mates and come up with an es­ti­mated score for each fa­cil­ity. This es­ti­mated score gave them a prior over de­tailed scores for each site- if the ex­perts all agreed that a site was a (0, 1, 2, 3), then that still im­plies a range over ac­tual val­ues. The eco­nomic loss re­sult­ing from a suc­cess­ful at­tack (3) could be any­where from $100 mil­lion to $1 billion. (No­tice that hav­ing a panel of ex­perts gave them a nat­u­ral way to de­ter­mine the spread of the prior be­yond the range in­her­ent in their an­swers- where the ex­perts agreed, they could clump the prob­a­bil­ity mass to­gether, with only a lit­tle on an­swers the ex­perts didn’t give, and where the ex­perts dis­agreed they knew where to spread the prob­a­bil­ity out over.) They already had, from an­other source, data on the effec­tive­ness of the risk re­duc­tions available at the var­i­ous sites and the costs of those re­duc­tions.

The high­est ac­tual con­se­quence elic­ited was for $6 billion, as­sum­ing a value of $6 mil­lion per life. The high­est VoI of get­ting a de­tailed site anal­y­sis, though, was only $1.1 mil­lion. From the defi­ni­tion, this shouldn’t be that sur­pris­ing- VoI is only large when you would be sur­prised or un­cer­tainty is high. For some sites, it was ob­vi­ous that DHS should in­vest in re­duc­ing risk; in oth­ers, it was ob­vi­ous that DHS shouldn’t in­vest in re­duc­ing risk. The de­tailed vuln­er­a­bil­ity anal­y­sis would just tell them what they already knew, and so wouldn’t provide any value. Some sites were on the edge- it might be worth­while to re­duce risk, it might not. For those sites, a de­tailed vuln­er­a­bil­ity anal­y­sis would provide value- but be­cause the site was on the edge, the ex­pected value of learn­ing more was nec­es­sar­ily small!6 Re­mem­ber, for VoI to be pos­i­tive you have to change your de­ci­sion, and if that doesn’t hap­pen there’s no VoI.

Distress­ingly, they went on to con­sider the case where risk re­duc­tion could not be performed with­out a de­tailed vuln­er­a­bil­ity anal­y­sis. Then, rather than mea­sur­ing VoI, they were mostly mea­sur­ing the value of risk re­duc­tion- and the max­i­mum value shot up to $840 mil­lion. When Bayesian ev­i­dence is good enough, re­quiring le­gal ev­i­dence can be costly.7

Med­i­cal Testing

About two years ago, I was sit­ting at my com­puter and no­ticed a black dot on my up­per arm. I idly scratched it, and then saw its lit­tle legs move.

It was an tick en­gorged on my blood, which I had prob­a­bly picked up walk­ing through the woods ear­lier. I re­moved it, then looked up on­line the proper way to re­move it. (That’s the wrong or­der, by the way: you need the in­for­ma­tion be­fore you make your de­ci­sion for it to be of any use. I didn’t do it the proper way, and thus in­creased my risk of dis­ease trans­mis­sion.)

Some ticks carry Lyme dis­ease, and so I looked into get­ting tested. I was sur­prised to learn that if I didn’t pre­sent any symp­toms by 30 days, the recom­men­da­tion was against test­ing. After a mo­ment’s re­flec­tion, this made sense- tests typ­i­cally have false pos­i­tive rates. If I didn’t have any symp­toms af­ter 30 days, even if I took the test and got a pos­i­tive re­sult the EV could be higher for no treat­ment than for treat­ment. In that case, the VoI of the test would be 0- re­gard­less of its out­come, I would have made the same de­ci­sion. If I saw symp­toms, though, then the test would be worth­while, as it could dis­t­in­guish Lyme dis­ease from an un­re­lated rash, headache, or fever. “Wait­ing for symp­toms to ap­pear” was the test with pos­i­tive VoI, not get­ting a blood test right away.

One could ar­gue that the blood test could have “peace of mind” value, but that’s dis­tinct from VoI. Even be­yond that, it’s not clear that you would get pos­i­tive peace of mind on net. Sup­pose the test has a 2% false pos­i­tive rate- what hap­pens when you mul­ti­ply the peace of mind from a true nega­tive by .98, and sub­tract the costs of deal­ing with the false pos­i­tives by .02? That could eas­ily be nega­tive.

(I re­main symp­tom-free; ei­ther the tick didn’t have Lyme dis­ease, didn’t trans­fer it to me, or my im­mune sys­tem man­aged to de­stroy it.)

Choos­ing a Career

Many ca­reers have sig­nifi­cant pre­req­ui­sites: if you want to be a doc­tor, you’re go­ing to have to go to med­i­cal school. Peo­ple of­ten have to choose where to in­vest their time with limited knowl­edge- you can’t know what the ca­reer prospects will be like when you grad­u­ate, how much you’ll en­joy your cho­sen field, and so on. Many peo­ple just choose based on ac­cu­mu­lated ex­pe­rience- lawyers were high-sta­tus and rich be­fore, so they sus­pect be­com­ing a lawyer now is a good idea.8

Re­duc­ing that un­cer­tainty can help you make a bet­ter de­ci­sion, and VoI helps de­cide what ways to re­duce un­cer­tainty are effec­tive. But this ex­am­ple also helps show the limits of VoI: VoI is best suited to situ­a­tions where you’ve done the back­ground re­search and are now con­sid­er­ing fur­ther ex­per­i­ments. With the bi­ased coin, we started off with a uniform prior; with the defen­sive in­vest­ments, we started off with es­ti­mated risks. Do we have a com­pa­rable spring­board for ca­reers?

If we do, it’ll take some build­ing. There’s a lot of differ­ent value func­tions we could build- it prob­a­bly ought to in­clude stress, in­come (both start­ing and life­time)9, risk of un­em­ploy­ment, satis­fac­tion, and sta­tus. It’s not clear how to elicit weights on those, though. There’s re­search on what makes peo­ple in gen­eral happy, but you might be un­com­fortable just us­ing those weights.10

There are also hun­dreds, if not thou­sands, of ca­reer op­tions available. Prior dis­tri­bu­tions on in­come are easy to find, but stress is harder to de­ter­mine. Unem­ploy­ment risk is hard to pre­dict over a life­time, es­pe­cially as it re­lies on macroe­co­nomic trends that may be hard to pre­dict. (The BLS pre­dicts em­ploy­ment num­bers out 10 years from data that’s a few years old. It seems un­likely that they’re set up to see crashes com­ing, though.)

Satis­fac­tion is prob­a­bly the eas­iest place to start: there are lots of ca­reer ap­ti­tude tests out there that can take self-re­ported per­son­al­ity fac­tors and turn that into a list of ca­reers you might be well-suited for. Now you have a man­age­able de­ci­sion prob­lem- prob­a­bly some­where be­tween six and twenty op­tions to re­search in depth.

What does that look like from a VoI frame­work? You’ve done a first screen­ing which has iden­ti­fied places where more in­for­ma­tion might al­ter your de­ci­sion. If you faint at the sight of blood, it doesn’t mat­ter how much sur­geons make, and so any time spent look­ing that up is wasted. If you do a quick scor­ing of the six value com­po­nents I listed above (af­ter brain­storm­ing for other things rele­vant to you), just weight­ing them with those quick val­ues may give you good pre­limi­nary re­sults. Only once you know what com­par­i­sons are rele­vant- “what trade­off be­tween sta­tus and un­em­ploy­ment risk am I will­ing to make?“—would you spend a long time nailing down your weights.

This is also a de­ci­sion prob­lem that could take a long, long time. (Even af­ter you’ve se­lected a ca­reer, the op­tion to switch is always pre­sent.) It can be use­ful to keep up­per and lower bounds for your es­ti­mates and up­date those along with your es­ti­mates- their cur­rent val­ues and their changes with the last few pieces of in­for­ma­tion you found can give you an idea of how much you can ex­pect to get from more re­search, and so you can finish re­search­ing and make a de­ci­sion at a care­fully cho­sen time, rather than when you get fa­tigued.

Conclusion

Let’s take an­other look at the defi­ni­tion: how much an­swer­ing a ques­tion al­lows a de­ci­sion-maker to im­prove its de­ci­sion.

The “an­swer­ing” is im­por­tant be­cause we need to con­sider all pos­si­ble an­swers.11 We’re re­plac­ing one ran­dom vari­able with two ran­dom vari­ables- in the case of the bi­ased coin, it re­placed one un­known coin (one flip) with ei­ther the lucky coin and the un­lucky coin (two flips- one to figure out which coin, one to bet on). When com­put­ing VoI, you can’t just con­sider one pos­si­ble an­swer, but all pos­si­ble an­swers con­sid­er­ing their rel­a­tive like­li­hood.12

The “im­prove” is im­por­tant be­cause VoI isn’t about sleep­ing bet­ter at night or cov­er­ing your ass. If you don’t ex­pect to change your de­ci­sion af­ter re­ceiv­ing this in­for­ma­tion, or you think that the ex­pected value of the in­for­ma­tion (the chance you change your de­ci­sion times the rel­a­tive value of the de­ci­sions) is lower than the cost of the in­for­ma­tion, just bite the bul­let and don’t run the test you were con­sid­er­ing.

The “de­ci­sion” is im­por­tant be­cause this isn’t just cu­ri­os­ity. Learn­ing facts is of­ten fun, but for it to fit into VoI some de­ci­sion has to de­pend on that fact. When watch­ing tele­vised poker, you know what all the hands are- and while that may al­ter your en­joy­ment of the hand, it won’t af­fect how any of the play­ers play. You shouldn’t pay much for that in­for­ma­tion, but the play­ers would pay quite a bit for it.13

1. Persi Di­a­co­nis pre­dicts most hu­man coin flips are fair to 2 dec­i­mals but not 3, and it’s pos­si­ble through train­ing to bias coins you flip. With a ma­chine, you can be pre­cise enough to get the coin to come up the same way ev­ery time.

2. There is one thing that isn’t coin-re­lated: your friend is offer­ing you this gam­ble, and prob­a­bly has in­for­ma­tion you don’t. That sug­gests the deal fa­vors him- but sup­pose that you and your friend just thought this up, and so nei­ther of you has more in­for­ma­tion than the other.

3. Your profit is 24p-12; your dis­tri­bu­tion on p is P(p)=2p, and so your dis­tri­bu­tion on profit is 48p2-24p in­te­grated from 0 to 1, which is 4.

4. Again, your profit is 24p-12; you have a uniform dis­tri­bu­tion on what I will tell you about p, but you only care about the sec­tion where p>.5. In­te­grated from .5 to 1, that’s 3.

5. When­ever elic­it­ing in­for­ma­tion from ex­perts, make sure to re­peat back to them what you heard and en­sure that they agree with it. You might know de­ci­sion the­ory, but the rea­son you’re talk­ing to ex­perts is be­cause they know things you don’t. Con­sis­tency can take a few iter­a­tions, and that’s to be ex­pected.

6. A com­mon trope in de­ci­sion anal­y­sis is “if a de­ci­sion is hard, flip a coin.” Most peo­ple balk at this be­cause it seems ar­bi­trary (and, more im­por­tantly, hard to jus­tify to oth­ers)- but if a de­ci­sion is hard, that typ­i­cally means both op­tions are roughly equally valuable, and so the loss from the coin flip com­ing up the wrong value is nec­es­sar­ily small.

7. That said, recom­men­da­tions for policy-mak­ers are hard to make here. Le­gal ev­i­dence is de­signed to be hard to game; Bayesian ev­i­dence isn’t, and so Bayesian ev­i­dence is only “good enough” if it’s not be­ing gamed. Check­ing your heuris­tic (i.e. the ex­pert’s es­ti­mates) to keep it hon­est can provide sig­nifi­cant value. Perform­ing de­tailed vuln­er­a­bil­ity anal­y­sis on some (how many?) ran­domly cho­sen sites for cal­ibra­tion is of­ten a good choice. Beyond that, I can’t do much be­sides point you to psy­chol­ogy to figure out good ways to di­ag­nose and re­duce bias.

8. It doesn’t ap­pear that this is the case any­more. The sup­ply of lawyers has dra­mat­i­cally in­creased, and so wages are de­clin­ing; as well, law is a pretty soul-crush­ing field from a stress, work-life bal­ance, and satis­fac­tion per­spec­tive. If law looks like the best field for you and you’re not in it for the money or sta­tus, the ad­vice I hear is to spe­cial­ize in a niche field that’ll put food on the table but stay in­ter­est­ing and tol­er­ably de­mand­ing.

9. Both of these cap­ture differ­ent in­for­ma­tion. A job with a high start­ing salary but no growth prospects might trans­late into more hap­piness than a job with a low start­ing salary but high growth prospects, for ex­am­ple.

10. Most of the hap­piness/​satis­fac­tion liter­a­ture I’ve seen has asked peo­ple about their at­tributes and their hap­piness/​satis­fac­tion. That’s not a ran­dom­ized trial, though, and so there could be mas­sive se­lec­tion effects. If we find that en­g­ineers are col­lec­tively less happy than wait­ers, does that mean en­g­ineer­ing causes un­hap­piness, un­hap­piness causes en­g­ineer­ing, that un­hap­piness and en­g­ineer­ing are caused by the same thing, or none of those?

11. Com­pare this with in­for­ma­tion the­ory, where bits are a prop­erty of an­swers, not ques­tions. Here, VoI is a prop­erty of ques­tions, not an­swers.

12. If you already know the cost of the in­for­ma­tion, then you can stop com­put­ing as soon as you find a pos­i­tive out­come good enough and likely enough that the VoI so far is higher than the cost.

13. In high-stakes poker games, the VoI can get rather high, and the de­ceit /​ read­ing in­volved is why poker is a more in­ter­est­ing game than, say, the lot­tery.