However, I was initially a bit confused by the section on EVPI. I think it is important, but it could be a lot clearer.
The expected opportunity loss (EOL) for a choice is the probability of the choice being “wrong” times the cost of it being wrong. So for example the EOL if the campaign is approved is $5M × 40% = $2M, and the EOL if the campaign is rejected is $40M × 60% = $24M.
The difference between EOL before and after a measurement is called the “expected value of information” (EVI).
It seems quite unclear what’s meant by “the difference between EOL before and after a measurement” (EOL of which option? is this in expectation?).
I think what must be intended is: your definition is for the EOL of an option. Now the EOL of a choice is the EOL of the option we choose given current beliefs. Then EVI is the expected reduction in EOL upon measurement.
Even this is more confusing than it often needs to be. At heart it’s the expected amount better you’ll do with the information. Sometimes you can factor out the EOL calculation entirely. For example say you’re betting $10 at even odds on a biased coin. You currently think there’s a 70% chance of it landing heads; more precisely you know it was either from a batch which lands heads 60% of the time, or from a batch which lands heads 80% of the time, but these are equiprobable. You could take a measurement to find out which batch it was from. Then you are certain that this measurement will change the EOL, but if you do it carefully the expected gain is equal to the expected loss, so there is no EVI. We could spot this directly because we know that whatever the answer is, we’ll bet on heads.
I think it might be useful to complete your simple example for EVPI (as in, this would have helped me to understand it faster, so may help others too):
Currently you’ll run the campaign, with EOL of $2M. With perfect information, you always choose the right option, so you expect the EOL to go down to 0. Hence the EVPI is $2M (this comes from the 40% of the time that the information stops you running the campaign and saving you $5M).
Then in the section on the more advanced model:
In this case, the EVPI turns out to be about $337,000. This means that we shouldn’t spend more than $337,000 to reduce our uncertainty about how many units will be sold as a result of the campaign.
Does this figure come from the book? It doesn’t come from the spreadsheet you linked to. By the way, there’s a mistake in the spreadsheet: when it assumes a uniform distribution it uses different bounds for two different parts of the calculation.
I like the coin example. In my experience the situation with clear choice is typical in small businesses. It often isn’t worth honing the valuation models for projects very long when it is very improbably that the presumed second best choice would turn out to be the best.
I guess the author is used to working for bigger companies that do everything in larger scale and thus have generally more options to choose from. Nothing untrue in the chapter but this point could have been pointed out.
Thanks, I liked this post.
However, I was initially a bit confused by the section on EVPI. I think it is important, but it could be a lot clearer.
It seems quite unclear what’s meant by “the difference between EOL before and after a measurement” (EOL of which option? is this in expectation?).
I think what must be intended is: your definition is for the EOL of an option. Now the EOL of a choice is the EOL of the option we choose given current beliefs. Then EVI is the expected reduction in EOL upon measurement.
Even this is more confusing than it often needs to be. At heart it’s the expected amount better you’ll do with the information. Sometimes you can factor out the EOL calculation entirely. For example say you’re betting $10 at even odds on a biased coin. You currently think there’s a 70% chance of it landing heads; more precisely you know it was either from a batch which lands heads 60% of the time, or from a batch which lands heads 80% of the time, but these are equiprobable. You could take a measurement to find out which batch it was from. Then you are certain that this measurement will change the EOL, but if you do it carefully the expected gain is equal to the expected loss, so there is no EVI. We could spot this directly because we know that whatever the answer is, we’ll bet on heads.
I think it might be useful to complete your simple example for EVPI (as in, this would have helped me to understand it faster, so may help others too): Currently you’ll run the campaign, with EOL of $2M. With perfect information, you always choose the right option, so you expect the EOL to go down to 0. Hence the EVPI is $2M (this comes from the 40% of the time that the information stops you running the campaign and saving you $5M).
Then in the section on the more advanced model:
Does this figure come from the book? It doesn’t come from the spreadsheet you linked to. By the way, there’s a mistake in the spreadsheet: when it assumes a uniform distribution it uses different bounds for two different parts of the calculation.
I like the coin example. In my experience the situation with clear choice is typical in small businesses. It often isn’t worth honing the valuation models for projects very long when it is very improbably that the presumed second best choice would turn out to be the best.
I guess the author is used to working for bigger companies that do everything in larger scale and thus have generally more options to choose from. Nothing untrue in the chapter but this point could have been pointed out.