I generally agree with the idea—a range prediction is much better than a binary prediction—but a normal prediction is not necessarily the best. It’s simple and easy to calculate, which is great, but it doesn’t account for edge cases.
Suppose you made the prediction about Biden a couple months before the election, and then he died. If he had been taken off of the ballot, he would have received zero votes, and even if he had been left on “Biden” would have received much fewer votes. Under your normal model, the chance of either of those happening is essentially zero, but there was probably a 1-5% chance of it happening. You can adjust for this by adding multiple normal curves and giving different weights to each curve, though I’m not sure how to do the scoring with this.
It also doesn’t work well for exponential behavior. For COVID cases in a given period, a few days difference in changing behavior could alter the number of deaths by a factor of 2 or more. That can be easily rectified though by putting your predictions in log form, but you have to remember to do that.
Overall though, normal predictions work well for most predictions, and we’d be better off using them!
Good Points, Everything is a conditional probability, so you can simply make conditional normal predictions:
Let A = Biden alive
Let B = Biden vote share
Pr(A)=Bernoulli(0.97)
p(B∣A)=N(0.54,0.03)
Then the normal probability is conditional on him being alive and does not count otherwise :)
Another solution is to make predictions from a T-distribution to get fatter tails. and then use “Advanced trick 3” to transform it back to a normal when calculating your calibration.
Given also the fact that many significant events seem to occur with on distributions with fat tails, assuming normal distributions may lead you to be systematically overconfident in your predictions. Though it’s still probably far, far better than using binary estimates.
You could make predictions from a t distribution to get fatter tails, but then the “easy math” for calibration becomes more scary… You can then take the “quartile” from the t distribution and ask what sigma in the normal that corresponds to. That is what I outlined/hinted at in the “Advanced Techniques 3”
I generally agree with the idea—a range prediction is much better than a binary prediction—but a normal prediction is not necessarily the best. It’s simple and easy to calculate, which is great, but it doesn’t account for edge cases.
Suppose you made the prediction about Biden a couple months before the election, and then he died. If he had been taken off of the ballot, he would have received zero votes, and even if he had been left on “Biden” would have received much fewer votes. Under your normal model, the chance of either of those happening is essentially zero, but there was probably a 1-5% chance of it happening. You can adjust for this by adding multiple normal curves and giving different weights to each curve, though I’m not sure how to do the scoring with this.
It also doesn’t work well for exponential behavior. For COVID cases in a given period, a few days difference in changing behavior could alter the number of deaths by a factor of 2 or more. That can be easily rectified though by putting your predictions in log form, but you have to remember to do that.
Overall though, normal predictions work well for most predictions, and we’d be better off using them!
Good Points, Everything is a conditional probability, so you can simply make conditional normal predictions:
Let A = Biden alive
Let B = Biden vote share
Pr(A)=Bernoulli(0.97)
p(B∣A)=N(0.54,0.03)
Then the normal probability is conditional on him being alive and does not count otherwise :)
Another solution is to make predictions from a T-distribution to get fatter tails. and then use “Advanced trick 3” to transform it back to a normal when calculating your calibration.
Given also the fact that many significant events seem to occur with on distributions with fat tails, assuming normal distributions may lead you to be systematically overconfident in your predictions. Though it’s still probably far, far better than using binary estimates.
You could make predictions from a t distribution to get fatter tails, but then the “easy math” for calibration becomes more scary… You can then take the “quartile” from the t distribution and ask what sigma in the normal that corresponds to. That is what I outlined/hinted at in the “Advanced Techniques 3”