Summary of my Participation in the Good Judgment Project

Fol­low-Up to Good Judg­ment Pro­ject, Sea­son Three.

Dur­ing the last fore­cast­ing sea­son I took part in the Good Judg­ment Pro­ject (GJP; see also the blog) and this is a short sum­mary of my par­ti­ci­pa­tion (ac­tu­ally trig­gered by ham­nox com­ment).

The GJP es­ti­mates world events like

  • Ukraine conflict

  • Arc­tic ice cap melting

  • Ebola out­break duration

  • Chi­nese sea conflict

  • ISIS attacks

  • Ter­ror­ist attacks

  • Oil price

  • Cer­tain ex­change rates

  • Elec­tion results

  • and many other poli­ti­cal events

To par­ti­ci­pate in that study one has to reg­ister (can’t re­mem­ber where ex­actly I stum­bled over the link, pos­si­bly the one at the top). And one has to do an prepara­tory on­line course and one has to pass an on­line test. At least I had to com­plete it. Whether the re­sult af­fected my as­sign­ment to any group I can’t say. The course ex­plains the scor­ing and gives recom­men­da­tions for mak­ing good fore­casts (choose fore­casts one has an edge in, es­ti­mate early, up­date of­ten, do post-mortems). The test seems to test for cal­ibra­tion and ac­cu­racy by ask­ing for known (mostly poli­ti­cal) events and whether one is sure about them.

The cur­rent fore­cast­ing sea­son started in Novem­ber 2014 and has just ended. I in­vested sig­nifi­cantly less then half an hour a week on 8 ques­tions of about 100 (and thus less than I pro­jected in an early ques­tion­aire). I did 2 to 15 up­dates for these ques­tions and I earned a score in the mid­dle range (mostly due to get­ting hit by an un­ex­pected ter­ror­ist at­tack). As I just learned I was as­signed to the study con­di­tion were I could nei­ther see the to­tal group es­ti­mate nor the es­ti­mates of the other group mem­bers—only their com­ments. I was some­what dis­ap­pointed by this as I had hoped to learn some­thing from how the scores de­vel­oped. Too bad I wasn’t in a pre­dic­tion marked group. But I hope to get the study re­sults later.

I will not take part in fur­ther rounds as I shy the effort for the types of fore­casts which are mostly poli­ti­cal. They are poli­ti­cal be­cause the spon­sor (guess who) is in­ter­ested mostly in poli­ti­cal events—less in eco­nom­i­cal, en­vi­ron­men­tal, sci­en­tific or other types. But I en­joyed fore­cast­ing ar­tic ice cap melt­ing and ebola—and net­ted a bet­ter than av­er­age score on that.

The scor­ing—at least in this group—is in­ter­est­ing and uses an av­er­aged Brier Score—av­er­aged over a) all fore­cast ques­tion­tion and b) within a ques­tion over all the days for which a fore­cast is pro­vided. I in­tended to game that by bet­ting on ques­tions that a) I could fore­cast well and b) that had an ex­pected re­li­able out­come. Sadly there were few of type a.

From this ex­pe­rience I learned that

  • such pre­dic­tion or­ga­ni­za­tions ask mostly for poli­ti­cal events,

  • poli­ti­cal events are hard to pre­dict and

  • pre­dict­ing poli­ti­cal events re­quires a lot of back­ground in­for­ma­tion.

  • I’m be­low av­er­age in pre­dict­ing poli­ti­cal event (at least com­pared to my group which I’d guess has more in­ter­est in poli­tics than I) but

  • I’m above av­er­age on non-poli­ti­cal top­ics.