However, I don’t think it’s actually always good advice when eliciting forecasts. For example, fairly often people ask whether or not they should make a question on Metaculus continuous or binary. Almost always my answer is “make it binary”. Binary questions get considerable more interest and are much easier to reason about. The additional value of having a more general estimate is almost always offset by:
Fewer predictors ⇒ less valuable forecast
People update less frequently ⇒ Stale forecast
Harder to visualize changes over time ⇒ Less engagement from the general public
I think your point 2. has been well dealt with by gbear605, but let me add my voice to his. Normal approximations are probably especially bad for lots of things we forecast. Metaculus uses a logistic distribution by default because it automatically includes slightly heavier tails than normal distributions.
Agreed 100% on 1) and with 2) I think my point is “start using the normal predictions as a gate way drug to over dispersed and model based predictions”
I stole the idea from Gelman and simplified it for the general community, I am mostly trying to raise the sanity waterline by spreading the gospel of predicting on the scale of the observed data.
All your critiques of normal forecasts are spot on.
Ideally everybody would use mixtures of over-dispersed distributions or models when making predictions to capture all sources of uncertainty
It is my hope that by educating people in continuous prediction the Metaculus trade off you mention will slowly start to favor the continuous predictions because people find it as easy as binary prediction… but this is probably a pipe dream, so I take your point
I think you’re advocating two things here:
Make a continuous forecast when forecasting a continuous variable
Use a normal distribution to approximate your continuous forecast
I think that 1. is an excellent tip in general for modelling. Here is Andrew Gelman making the same point
However, I don’t think it’s actually always good advice when eliciting forecasts. For example, fairly often people ask whether or not they should make a question on Metaculus continuous or binary. Almost always my answer is “make it binary”. Binary questions get considerable more interest and are much easier to reason about. The additional value of having a more general estimate is almost always offset by:
Fewer predictors ⇒ less valuable forecast
People update less frequently ⇒ Stale forecast
Harder to visualize changes over time ⇒ Less engagement from the general public
I think your point 2. has been well dealt with by gbear605, but let me add my voice to his. Normal approximations are probably especially bad for lots of things we forecast. Metaculus uses a logistic distribution by default because it automatically includes slightly heavier tails than normal distributions.
Agreed 100% on 1) and with 2) I think my point is “start using the normal predictions as a gate way drug to over dispersed and model based predictions”
I stole the idea from Gelman and simplified it for the general community, I am mostly trying to raise the sanity waterline by spreading the gospel of predicting on the scale of the observed data. All your critiques of normal forecasts are spot on.
Ideally everybody would use mixtures of over-dispersed distributions or models when making predictions to capture all sources of uncertainty
It is my hope that by educating people in continuous prediction the Metaculus trade off you mention will slowly start to favor the continuous predictions because people find it as easy as binary prediction… but this is probably a pipe dream, so I take your point