I don’t know if there’s a name for this other than regression-to-the-mean, but here’s a reminder of a useful principle to keep in mind with all things like this: if you evaluate a number of options and pick the one that seems the best, you should expect to be disappointed.
Suppose GiveWell evaluates 10 charities and tells you its estimate of cost-per-life-saved for each. With no extra information, donating to the highest ranked charity is your best option. BUT, if the charities are at all competitive with each other (relative to the uncertainty in cost-per-life estimates), then the top charity is most likely to be over estimated. GiveWell has reported the true value plus some error and so the error is most likely large for the charity with the best reported estimate. Therefore you should expect to be disappointed in the choice.
So I am not surprised that their top charities are routinely found to have an underestimated cost-per-life.
That will tell you that you shouldn’t expect to spend more to save one life than estimated from the top charity; it might even tell you that the expected cost of the top rated charity is higher than the estimated cost from the next-ranked charity.
Without a reason to believe that the accuracy of the estimate for the lower-ranked charity is greater, you can’t conclude that the expected cost of the higher-ranked charity is greater than the expected cost of the lower-ranked charity.
I don’t see any need to “solve” the “problem”, at least in this context. The goal is maximize utility, not to minimize “surprise”. All we care about is the ordinal values of the choices; getting the quantitative values right is simply a means to that end. The “solution” presented in your article doesn’t make any sense; unless there’s some asymmetry between the choices, how can your “solution” change the ordinal value of the choices?
Yes, but the exact values for dollars per life saved given by GiveWell are both relevant in the article we’re discussing and is also frequently cited here on LessWrong.
Doesn’t this imply that it makes sense to donate to more than one charity? Consider just the top three charities; there are 3! ways to rank them and associated probabilities of each of those rankings being accurate. Say there’s a 90% probability that the 1st ranked charity is actually the most efficient, a 9% probability that the 2nd ranked charity is the most efficient, and a ~1% probability that the 3rd ranked charity is the most efficient. To me it makes sense to send 0.9X to charity #1, 0.09X to charity #2, and 0.01X to charity #3, where X is the amount of money available for charitable donations.
Not even close. Imagine if, instead of charities, these are colored balls. And instead of altruistic benefit, you’re getting paid (or money gets sent to your charity of choice). Say I gave you $10, and you get a return of 200:1 on any money placed on the ball that comes out. How do you distribute your money? Any distribution other than all on the most likely loses out.
Alternatively, imagine colored cards in a deck. You guess what color comes next, and you get $10 for every correct guess. What do you guess, assuming cards are replaced every time? In a hundred guesses, do you change your guess 10 times? If you do you’ll lose out.
If you split the donations in this way, you are lowering the expected money donated to the most efficient charity: it’s 0.9X*0.9 + 0.09X*0.09 + 0.01X*0.01, for a total of 0.8182X. By donating X to the 1st ranked charity, you donate 0.9X to the most efficient charity in expectation.
Actually, the ranking of the charities is irrelevant, as is the question of which charity is the most efficient; it’s only the absolute efficiency of your donation that matters. But if you look at that metric instead, the same problem occurs.
To put it another way: it’s almost certain that the ranking of the top charity is inflated; and it may even be almost certain that some of the other charities are better. However, no single charity is more likely to be good than the top charity. For each dollar donated, the single best place to send it is the top-ranked charity, and if you split your donations, that means that some of your dollars are going to a charity where they’re less likely to do good.
The answer is no, for reasons that are hard to articulate succinctly. But the fact that the variance in “actual” expected value is lower than it initially appears makes “donating to learn” more attractive, and for this reason, among others, GiveWell recommended that donors split their donations between its top charities.
I don’t know if there’s a name for this other than regression-to-the-mean, but here’s a reminder of a useful principle to keep in mind with all things like this: if you evaluate a number of options and pick the one that seems the best, you should expect to be disappointed.
Suppose GiveWell evaluates 10 charities and tells you its estimate of cost-per-life-saved for each. With no extra information, donating to the highest ranked charity is your best option. BUT, if the charities are at all competitive with each other (relative to the uncertainty in cost-per-life estimates), then the top charity is most likely to be over estimated. GiveWell has reported the true value plus some error and so the error is most likely large for the charity with the best reported estimate. Therefore you should expect to be disappointed in the choice.
So I am not surprised that their top charities are routinely found to have an underestimated cost-per-life.
This is called the optimizer’s curse. There are standard ways to solve the problem, e.g. multilevel modeling.
Good to know, thanks!
That will tell you that you shouldn’t expect to spend more to save one life than estimated from the top charity; it might even tell you that the expected cost of the top rated charity is higher than the estimated cost from the next-ranked charity.
Without a reason to believe that the accuracy of the estimate for the lower-ranked charity is greater, you can’t conclude that the expected cost of the higher-ranked charity is greater than the expected cost of the lower-ranked charity.
I don’t see any need to “solve” the “problem”, at least in this context. The goal is maximize utility, not to minimize “surprise”. All we care about is the ordinal values of the choices; getting the quantitative values right is simply a means to that end. The “solution” presented in your article doesn’t make any sense; unless there’s some asymmetry between the choices, how can your “solution” change the ordinal value of the choices?
Yes, but the exact values for dollars per life saved given by GiveWell are both relevant in the article we’re discussing and is also frequently cited here on LessWrong.
Doesn’t this imply that it makes sense to donate to more than one charity? Consider just the top three charities; there are 3! ways to rank them and associated probabilities of each of those rankings being accurate. Say there’s a 90% probability that the 1st ranked charity is actually the most efficient, a 9% probability that the 2nd ranked charity is the most efficient, and a ~1% probability that the 3rd ranked charity is the most efficient. To me it makes sense to send 0.9X to charity #1, 0.09X to charity #2, and 0.01X to charity #3, where X is the amount of money available for charitable donations.
Not even close. Imagine if, instead of charities, these are colored balls. And instead of altruistic benefit, you’re getting paid (or money gets sent to your charity of choice). Say I gave you $10, and you get a return of 200:1 on any money placed on the ball that comes out. How do you distribute your money? Any distribution other than all on the most likely loses out.
Alternatively, imagine colored cards in a deck. You guess what color comes next, and you get $10 for every correct guess. What do you guess, assuming cards are replaced every time? In a hundred guesses, do you change your guess 10 times? If you do you’ll lose out.
If you split the donations in this way, you are lowering the expected money donated to the most efficient charity: it’s 0.9X*0.9 + 0.09X*0.09 + 0.01X*0.01, for a total of 0.8182X. By donating X to the 1st ranked charity, you donate 0.9X to the most efficient charity in expectation.
Actually, the ranking of the charities is irrelevant, as is the question of which charity is the most efficient; it’s only the absolute efficiency of your donation that matters. But if you look at that metric instead, the same problem occurs.
To put it another way: it’s almost certain that the ranking of the top charity is inflated; and it may even be almost certain that some of the other charities are better. However, no single charity is more likely to be good than the top charity. For each dollar donated, the single best place to send it is the top-ranked charity, and if you split your donations, that means that some of your dollars are going to a charity where they’re less likely to do good.
The answer is no, for reasons that are hard to articulate succinctly. But the fact that the variance in “actual” expected value is lower than it initially appears makes “donating to learn” more attractive, and for this reason, among others, GiveWell recommended that donors split their donations between its top charities.
Yes, this is what I meant by regression to the mean.