I don’t think this is an especially good idea for a bunch of reasons:
It’s hard enough getting pundits to put numbers on their forecasts,
adding a bunch of additional demands on them seems counterproductive
making their forecasting competitive may put them off
It’s extremely unlikely they’ll answer enough questions whereby you can statistically significantly tell the difference between them (unless they are really terrible)
Ideally Metaculus (or other prediction platforms) should be asking sufficiently many interesting questions about future years that the questions which the pundits choose to forecast on are already predicted on, and we can make comparisons from there.
I would recommend this article from the EA forum which also lays out a bunch of additional issues around prediction contests
That’s a great point. [Getting more pundits to make predictions at all] is much more valuable than [more accurately comparing pundits who do make predictions] right now, to such an extent that I now doubt whether my idea was worthwhile.
I don’t think this is an especially good idea for a bunch of reasons:
It’s hard enough getting pundits to put numbers on their forecasts,
adding a bunch of additional demands on them seems counterproductive
making their forecasting competitive may put them off
It’s extremely unlikely they’ll answer enough questions whereby you can statistically significantly tell the difference between them (unless they are really terrible)
Ideally Metaculus (or other prediction platforms) should be asking sufficiently many interesting questions about future years that the questions which the pundits choose to forecast on are already predicted on, and we can make comparisons from there.
I would recommend this article from the EA forum which also lays out a bunch of additional issues around prediction contests
That’s a great point. [Getting more pundits to make predictions at all] is much more valuable than [more accurately comparing pundits who do make predictions] right now, to such an extent that I now doubt whether my idea was worthwhile.
This feels solvable with a sufficiently large monetary prize.