I think I misunderstand the question, or I don’t get the assumptions, or I’ve gone terribly wrong.
Let me see if I’ve got the problem right to begin with. (I might not.)
40% of baseball players hit over 10 home runs a season. (I am making this up.)
Joe is a baseball player.
Baseball projector Mayne says Joe has a 70% chance of hitting more than 10 home runs next season. Baseball projector Szymborski says Joe has an 80% chance of hitting more than10 home runs next season. Both Mayne and Szymborski are aware of the usual rate of baseball players hitting more than 10 home runs.
Is this the problem?
Because if it is, the use of the prior is wrong. If the experts know the prior, and we believe the experts, the prior’s irrelevant—our odds are 75%.
There are a lot of these situations in which regression to the mean, use of averages in determinations, and other factors are needed. But in this situation, if we assume reasonable experts who are aware of the general rules, and we value those experts’ opinions highly enough, we should just ignore the prior—the experts have already factored that in. When Nate Silver gives you the odds that Barack Obama wins the election, you shouldn’t be factoring in P(Incumbent wins) or anything else—the cake is prebaked with that information.
Since this rejects a strong claim in the post, it’s possible I’m very seriously misreading the problem. Caveat emptor.
You’re reading it correctly, but I disagree with your conclusion. If Mayne says p=.7, and Szymborski says p=.8, and their estimates are independent—remember, my classifiers are not human experts, they are not correlated—then the final result must be greater than .8. You already thought p=.8 after hearing Szymborski. Mayne’s additional opinion says Joe is more-likely than average to hit more than 10 home runs, and is based on completely different information than Szymborski’s, so it should make Joe’s chances increase, not decrease.
remember, my classifiers are not human experts, they are not correlated
Is that necessarily true? It seems that it should depend on whether they have underlying similarities (eg a similar systematic bias) in their algorithms.
I think I misunderstand the question, or I don’t get the assumptions, or I’ve gone terribly wrong.
Let me see if I’ve got the problem right to begin with. (I might not.)
40% of baseball players hit over 10 home runs a season. (I am making this up.)
Joe is a baseball player.
Baseball projector Mayne says Joe has a 70% chance of hitting more than 10 home runs next season. Baseball projector Szymborski says Joe has an 80% chance of hitting more than10 home runs next season. Both Mayne and Szymborski are aware of the usual rate of baseball players hitting more than 10 home runs.
Is this the problem?
Because if it is, the use of the prior is wrong. If the experts know the prior, and we believe the experts, the prior’s irrelevant—our odds are 75%.
There are a lot of these situations in which regression to the mean, use of averages in determinations, and other factors are needed. But in this situation, if we assume reasonable experts who are aware of the general rules, and we value those experts’ opinions highly enough, we should just ignore the prior—the experts have already factored that in. When Nate Silver gives you the odds that Barack Obama wins the election, you shouldn’t be factoring in P(Incumbent wins) or anything else—the cake is prebaked with that information.
Since this rejects a strong claim in the post, it’s possible I’m very seriously misreading the problem. Caveat emptor.
You’re reading it correctly, but I disagree with your conclusion. If Mayne says p=.7, and Szymborski says p=.8, and their estimates are independent—remember, my classifiers are not human experts, they are not correlated—then the final result must be greater than .8. You already thought p=.8 after hearing Szymborski. Mayne’s additional opinion says Joe is more-likely than average to hit more than 10 home runs, and is based on completely different information than Szymborski’s, so it should make Joe’s chances increase, not decrease.
Is that necessarily true? It seems that it should depend on whether they have underlying similarities (eg a similar systematic bias) in their algorithms.