GJP seems to be proving that, with actual quantified data; excerpting from a recent email I received as a participant:
Phil Tetlock has been recording a “subject matter expert” stream of forecasts, based on his review of all of the data we collect from you PLUS the input of subject-matter-experts whom he consults. [...] The “expert” forecast stream has fared far worse this season, with a current overall Brier score of .31 [...] The Good Judgment Team as a whole is still outperforming the benchmark forecasts against which IARPA judges us. The current overall Brier score for our best aggregate forecast stream is .2554. By comparison, the official “par” (IARPA’s benchmark) has risen to .48. We’re beating “par” by 61%. [...] Several of you have achieved amazing accuracy this season. (Any individual or team whose current Brier score is lower than the .31 for the “expert” stream should feel very proud.)
My current score is .33, fairly close to the “expert” stream score, so with some basic notions and Google it’s possible, apparently, to do nearly as well as “experts”. (The team I’m on has a so-so aggregate score, only .41 - I assume that the best teams are made up of reasonably good forecasters that are also good at pooling their work.)
The study is ongoing. It’s an effort to have “normal” people make predictions on world events, assessing individual-, team- and project-level performance as Brier scores.
GJP seems to be proving that, with actual quantified data; excerpting from a recent email I received as a participant:
My current score is .33, fairly close to the “expert” stream score, so with some basic notions and Google it’s possible, apparently, to do nearly as well as “experts”. (The team I’m on has a so-so aggregate score, only .41 - I assume that the best teams are made up of reasonably good forecasters that are also good at pooling their work.)
What’s GJP? Any links to info on this study and what the various numbers mean?
Good Judgment Project, see also my recent two-part post.
The study is ongoing. It’s an effort to have “normal” people make predictions on world events, assessing individual-, team- and project-level performance as Brier scores.