Introducing Foretold.io: A New Open-Source Prediction Registry

I’m happy to an­nounce a semi-pub­lic beta of Fore­told.io for the EA/​LessWrong com­mu­nity. I’ve spent much of the last year work­ing on cod­ing & de­vel­op­ment, with lots of help by Ja­cob Lager­ros on product and scor­ing de­sign. Spe­cial thanks to the Long-Term Fu­ture Fund and its donors, who’s con­tri­bu­tion to the pro­ject helped us to hire con­trac­tors to do much of the en­g­ineer­ing & de­sign.

You can use Fore­told.io right away by fol­low­ing this link. Cur­rently pub­lic ac­tivity is only shown to logged in users, but I ex­pect that to be opened up over the next few weeks. There are cur­rently only a few pub­lic com­mu­ni­ties of pre­dictable ques­tions, but that will change over time.

The Main Concept

We aim for Fore­told.io to be use­ful as a gen­eral-pur­pose pre­dic­tion reg­istry, with the po­ten­tial to be used for more spe­cific pre­dic­tion pur­poses.

The main fea­tures of a pre­dic­tion reg­istry in­clude things like:

  • Peo­ple can spec­ify ques­tions to be predicted

  • Fore­cast­ers can pre­dict those questions

  • Ques­tions can ei­ther be re­solved with an­swers or cancelled

  • After ques­tions are re­solved, fore­cast­ers are scored on mean­ingful metrics

In ad­di­tion to the es­sen­tials, we fo­cused on some other use­ful fea­tures in­clud­ing:

Full dis­tri­bu­tion fore­casts for con­tin­u­ous vari­ables
In Fore­told.io, vari­ables are es­ti­mated with ar­bi­trary prob­a­bil­ity dis­tri­bu­tions. Most ex­ist­ing fore­cast­ing tools only al­low for bi­nary and cat­e­gor­i­cal bi­nary ques­tions, or rel­a­tively sim­ple dis­tri­bu­tions. Fore­told.io saves ar­bi­trary cu­mu­la­tive den­sity func­tions. The main in­put ed­i­tor is a fork of that in Guessti­mate. We plan to add more in­put meth­ods in the fu­ture.

“Com­mu­ni­ties” with cus­tom pri­vacy set­tings
Fore­told.io al­lows for groups to col­lab­o­rate on fore­cast­ing differ­ent sets of ques­tions. Com­mu­ni­ties can be pub­lic or pri­vate, and ques­tion cre­ators can eas­ily move their ques­tions be­tween com­mu­ni­ties. I’ve talked to sev­eral Effec­tive Altru­ist or­ga­ni­za­tions that have in­ter­nal fore­cast­ing se­tups, but al­most all use in-house solu­tions with Google Docs. One of the main bot­tle­necks seems to be easy pri­vate com­mu­nity sup­port.

A GraphQL API, with sup­port for bots
Users can cre­ate bots that get scored in­di­vi­d­u­ally. They can use the same GraphQL API that the Fore­told.io client uses. You can see in­for­ma­tion about how to use the API here. This part is still early, but will con­tinue to im­prove.

In the fu­ture we hope that the API will be used to do things like:

  • Make forecasts

  • Make & re­solve questions

  • Au­to­mate the setup of new pre­dic­tion experiments

  • Make dash­boards of use­ful forecasts

In­tended Uses

Similar to Guessti­mate, Fore­told.io it­self is not do­main-spe­cific. It could be used in mul­ti­ple kinds of se­tups; for in­stance, for per­sonal use, group use, or for a siz­able open pre­dic­tion tour­na­ment. Hope­fully over the com­ing years we’ll iden­tify which spe­cific uses and se­tups are the most promis­ing and op­ti­mize ac­cord­ingly.

Re­cently it’s been used for:

  • Var­i­ous per­sonal/​in­di­vi­d­ual questions

  • In­ter­nal group pre­dic­tions at FHI

  • A cur­rently-open tour­na­ment on pre­dict­ing the up­com­ing EA sur­vey responses

  • A few small fore­cast­ing experiments

We en­courage broad ex­per­i­men­ta­tion. Feel free to make as many pub­lic & pri­vate com­mu­ni­ties as you like for differ­ent pur­poses. If you’d be in­ter­ested in dis­cussing pos­si­ble de­tails, please reach out.

Questions

I’m in­ter­ested in perform­ing an ex­per­i­ment that could use the track­ing of prob­a­bil­ity dis­tri­bu­tions. Can I use Fore­told.io?
Yes! Fore­told.io is open-source, and we’re very happy to give spe­cial sup­port to re­searchers and similar in­ter­ested in work­ing with prob­a­bil­ity dis­tri­bu­tions and/​or fore­casts. It’s made to be rea­son­ably gen­eral-pur­pose and ex­tend­able via the API.

To get started, sim­ply cre­ate a com­mu­nity on Fore­told.io and make a few ques­tions. If you pre­fer, you can also fork the code­base and run the app sep­a­rately.

Are you co­or­di­nat­ing with other fore­cast­ing pro­jects?
In the last few years sev­eral efforts and re­search pro­jects around “fore­cast­ing” have emerged, speci­fi­cally around AI. Most of these are fo­cused on do­main-spe­cific re­search, rather than tech­ni­cal in­fras­truc­ture. I have been talk­ing with sev­eral of the other groups, and have been work­ing par­tic­u­larly closely with Ben Gold­haber and Ja­cob Lager­ros of Par­allel Fore­cast.

Why not just part­ner with an ex­ist­ing tech­ni­cal fore­cast­ing reg­istry and add fea­tures to that?
In gen­eral, I’ve found that it’s re­ally hard to join a group and get them to dra­mat­i­cally change their pri­ori­ties. Many of the new ad­di­tions in Fore­told.io are pretty sig­nifi­cant, and the roadmap is am­bi­tious.

Is there any con­nec­tion be­tween Fore­told.io and Guessti­mate?
Fore­told.io uses a fork of the dis­tri­bu­tion ed­i­tor from Guessti­mate. The dis­tri­bu­tion syn­tax is the same (“5 to 20”). In the fu­ture we plan to make it easy to im­port Fore­told.io vari­ables into Guessti­mate, and to use Guessti­mate vari­ables for pre­dic­tions in Fore­told.io.

Details

Tech­ni­cal de­tails
Fore­told.io uses Node.js and Ex­press.js with Apollo for the GraphQL server, and Rea­sonML and Re­act for the client. The database is Post­greSQL. The ap­pli­ca­tion is cur­rently hosted on Heroku.

Fund­ing
The pro­ject has raised $90,000 from the Long-Term Fu­ture Fund. Around $25,000 of that has been spent so far, mostly on pro­gram­ming and de­sign help.

Own­er­ship
Fore­told.io is open source. In the fu­ture, I in­tend for it to be sup­ported via a non­profit.

Get In­volved
Fore­told.io is free & open to use of all (le­gal) kinds. That said, if you in­tend to make se­ri­ous use of the API, please let me know be­fore­hand.

If you’re in­ter­ested in col­lab­o­rat­ing on ei­ther the plat­form, for­mal ex­per­i­ments, or re­lated re­search, please reach out to me, ei­ther via pri­vate mes­sage or email (ozziea­gooen@gmail.com). I’m par­tic­u­larly look­ing for en­g­ineers and peo­ple who want to set up fore­cast­ing tour­na­ments on im­por­tant top­ics.

Select Screenshots

In­dex View
index-view

Ques­tion View
https://i.ibb.co/2gNSFtc/image.png


Many thanks to Ja­cob Lager­ros, Ondřej Ba­j­gar, and Rose Had­shar for sev­eral use­ful com­ments on this post