[Job ad] Lead an ambitious COVID-19 forecasting project [Deadline extended: June 10th]

In brief: I’m hiring for a pro­ject man­ager for Epi­demic Fore­cast­ing, an in­de­pen­dent pro­ject spun off from the Fu­ture of Hu­man­ity In­sti­tute that pro­vides fore­cast­ing in­for­ma­tion to gov­ern­ments and other de­ci­sion-mak­ers dur­ing the covid pan­demic. [Ap­pli­ca­tion dead­line ex­tended to June 10th]. You need to have good judge­ment, the abil­ity to work fast and hard lead­ing a team, and refer­ences from peo­ple I trust. You need to be able to com­mit to 3 months of full-time paid work, and there’s the po­ten­tial to scale up af­ter­wards to a full-time or­gani­sa­tion if you find trac­tion.

Pri­vate mes­sage me to talk more. For the rest of the post, I’ll give more info on the pro­ject, what’s in­volved and why you might want to work on this.

What is the state of this pro­ject?

Since the be­gin­ning of March, I’ve led the Epi­demic Fore­cast­ing pro­ject which spun off from the Fu­ture of Hu­man­ity In­sti­tute at Oxford.

So far we’ve:

  • Pro­vided policy analy­ses for se­nior health offi­cials of re­gions with tens of mil­lions inhabitants

  • Ad­vised a ma­jor vac­cine can­di­date on where to lo­cate tri­als with as many as 100,000 par­ti­ci­pants

  • Built the in­ter­ac­tive tools at Epi­demicFore­cast­ing.org, which served 10k users/​​day at its peak

  • Worked with an in-house team of ex­pert fore­cast­ers to pro­duce >3000 fore­casts on ques­tions of im­prove­ments to test­ing ca­pac­ity and con­tact trac­ing, herd im­mu­nity, fur­ther lock­down, and more

  • Built soft­ware in­fras­truc­ture to in­te­grate those fore­casts into epi­demiolog­i­cal mod­el­ling soft­ware like GleamViz and run simu­la­tions at scale

  • Built one of the most thor­oughly val­i­dated Bayesian mod­els of the effects of coun­ter­mea­sures we are aware of

Over the last 3 months we’ve found that this pro­ject has real trac­tion and de­mand, and I think it’s time for a fo­cused team to com­mit to at least an­other 3-12 months on this, and ei­ther wrap up once op­por­tu­ni­ties go away, or build it into a non-profit/​for-profit that serves an im­por­tant long-term func­tion for miti­gat­ing pan­demics.

This post is an in­vi­ta­tion for a new pro­ject man­ager to step up. I’m go­ing to pur­sue other pro­jects that I think have more long-term po­ten­tial for me per­son­ally. If I find some­one who can do the man­age­ment work, can de­liver out­put fast, and has the sound judg­ment needed to take over, then I will help you get started with the team, in­tro­duce you to ma­jor re­la­tion­ships (gov­ern­ments, vac­cine com­pa­nies, and fun­ders) and help ad­vise/​men­tor you over the com­ing month. If I don’t I will start wind­ing down my re­spon­si­bil­ities and in­volve­ment.

Why would this pro­ject be valuable?

Peter Thiel says the most suc­cess­ful pro­jects are based on se­crets that no­body else has re­al­ised. This seems true to me. Here is my sense of the se­crets this pro­ject is based around.

  • There is a se­vere short­age of epi­demiolog­i­cal mod­el­ling abil­ity globally. There is not enough sup­ply to meet the sud­den, mas­sive in­crease in de­mand, from ev­ery coun­try in the world and thou­sands of large or­gani­sa­tions, fol­low­ing covid. For ex­am­ple, we’ve spo­ken to de­ci­sion-mak­ers gov­ern­ing tens of mil­lions of peo­ple, who use no epi­demiolog­i­cal mod­el­ling to sup­port their de­ci­sion-mak­ing. At the same time, most of the ac­tual re­sources are mas­sively skewed to­wards fore­cast­ing de­vel­op­ments in high-in­come coun­tries.

  • This short­age can par­tially be filled by good gen­er­al­ist fore­cast­ers, who are available in greater sup­ply than trained epi­demiol­o­gists. (See McAn­drew (2020) for ev­i­dence of fore­cast­ers out­perform­ing do­main ex­perts in pre­dict­ing covid. Far­row et al. (2017) gives more de­tails on us­ing hu­man judge­ment for epi­demiolog­i­cal fore­cast­ing, and Mel­lers et al. (2015) is a good re­view of Su­perfore­cast­ing.)

  • There are ma­jor prob­lems with epi­demiolog­i­cal mod­el­ling as usu­ally prac­ticed, that are well placed to ad­dress by re­searchers who have thought sen­si­bly about tech­nol­ogy and rea­son­ing un­der un­cer­tainty in gen­eral. For ex­am­ple, we’ve spo­ken to de­ci­sion-mak­ers who sim­ply ex­trap­o­late SIR mod­els to 2021 and use that as a key in­put into their de­ci­sions, with­out re­gard for the fact that on such timelines the ma­jor­ity of var­i­ance in out­come is driven by how gov­ern­ments will re­spond, how peo­ple will re­spond, and how tech­nol­ogy will change (e.g. with im­proved con­tact trac­ing or test­ing ca­pac­ity).

  • It is pos­si­ble to ex­trap­o­late from limited data based on good judge­ment (in the su­perfore­caster-sense), but this is barely util­ised in much of poli­cy­mak­ing. Many de­ci­sion-mak­ers are mak­ing policy based on con­firmed cases num­bers (even in low-and-mid­dle-in­come coun­tries with very limited test­ing). Even a rough ap­pli­ca­tion of ba­sic fore­cast­ing from some­one with a track-record on Me­tac­u­lus or Good Judge­ment would provide a much bet­ter baseline. As an­other ex­am­ple, we’ve en­coun­tered se­nior re­searchers hand­ing de­ci­sion-mak­ers com­plex mod­els, but be­ing very re­luc­tant to set the in­put pa­ram­e­ters due to only hav­ing limited data, in­stead ex­pect­ing the de­ci­sion-mak­ers to do that(!) The method­ol­ogy of us­ing fore­caster judge­ment in the face of scarce data has not yet been im­ple­mented across all the rele­vant fields (even though some groups are try­ing).

My sense is that, in line with what Scott Alexan­der says about failures dur­ing the covid pan­demic, you can sub­stan­tially im­prove the de­ci­sion-mak­ing of many of hu­man­ity’s best in­sti­tu­tions by us­ing the ba­sic skill of rea­son­ing un­der un­cer­tainty and be­ing will­ing to move fast and act quickly. You do need to be able to quickly make se­ri­ous qual­ity checks on a fore­cast (e.g. “What does this num­ber re­ally im­ply about the next 2 months of growth of the dis­ease in coun­try X?”), but you do not need years of ex­per­tise in this area to pick it up, es­pe­cially when you have peo­ple around you do­ing a lot of the heavy lift­ing who aren’t do­ing the man­age­ment (and aren’t re­spon­si­ble for the fi­nal de­ci­sions).

Where can this pro­ject go?

Like many star­tups, the world should de­ter­mine the product you make, not you. While I don’t know where this will end up, here are some am­bi­tious win con­di­tions.

  • Cause the date by which we have a vac­cine to move sev­eral weeks for­ward by pre­vent­ing failed effi­cacy tri­als.

  • Spin­ning off our mod­el­ling in­fras­truc­ture into open-source tool­ing for other re­searchers.

  • Serve as co­or­di­na­tion point con­nect­ing policy-mak­ers to the right epi­demiolog­i­cal mod­el­lers and fore­cast­ers, and re­bal­anc­ing the cur­rent mas­sive mis­al­lo­ca­tion be­tween high and low-in­come coun­tries. (I think there are 100+ mod­el­lers who could have vastly more im­pact if they tar­geted coun­tries more wisely, rather than mak­ing yet an­other marginal con­tri­bu­tion to US/​Europe mod­el­ling, and this pro­ject could play a sub­stan­tial role in mak­ing that hap­pen.)

  • Giv­ing state-of-the-art epi­demic mod­el­ling tech to many or­ders of mag­ni­tude more peo­ple than cur­rently by main­tain­ing pub­lic dash­boards and on­line tools (e.g. look at cur­rent epi­demicfore­cast­ing.org and make it 10x as use­ful and in­tu­itive). The win here is both in­form­ing de­ci­sion-mak­ers and the pub­lic.

  • Build long-term re­la­tions with de­ci­sion-mak­ers to offer policy ad­vice for fu­ture pan­demics.

  • ...or some as-of-yet un­known pivot that will be clear to the team in a few months

Thus far there have been many op­por­tu­ni­ties to pivot. We’ve moved from build­ing pub­lic dash­boards, to policy con­sult­ing, to vac­cine trial port­fo­lio op­ti­mi­sa­tion.

There are a wealth of im­por­tant prob­lems that need solv­ing, and we have some key skills al­low­ing us to do it. There will be many op­por­tu­ni­ties for the next pro­ject man­ager to figure out how to nav­i­gate this space.

What’s in­volved? What skills do I need?

Here are some of the kinds of tasks I’ve worked on, many of which you would do too:

  • Writ­ing policy re­ports (mak­ing com­plex es­ti­mates sim­ple and clear)

  • Giv­ing in­put on UI de­sign for websites

  • Writ­ing ques­tions for Su­perfore­cast­ers to answer

  • Run­ning meet­ings and un­der­stand­ing the needs of clients

More gen­er­ally, you need to be able to:

  • Be the glue that con­nects differ­ent teams

  • Learn how to man­age mod­el­lers run­ning epi­demiolog­i­cal mod­els us­ing soft­ware like GleamViz

  • Ac­tu­ally ship products

  • Deal with wholly re­mote work

  • Step up to work­ing 70+ hour weeks in pe­ri­ods when it’s called for, and know how to man­age your san­ity dur­ing these times (i.e. know how to take real rest).

You need to be some­one that some­one I trust trusts.

What ex­actly am I sign­ing up for?

Com­mit­ting to run this pro­ject full-time, salaried, for at least 3 months.

The world is strug­gling to meet the over­whelming de­mand for covid fore­cast­ing and anal­y­sis. You’re in a po­si­tion to im­prove the de­ci­sions they make about covid, and save thou­sands of lives, if you want to put in the work. We’ve set up a team with some good skills and re­la­tion­ships. The high­est var­i­ance part has already hap­pened, where we didn’t even know if there was de­mand. There is de­mand.

If you’re some­one who can ex­e­cute and have shown good judg­ment to me or some­one I trust (e.g. Ben Pace, Oliver Habryka) then I’m in­ter­ested in hand­ing the reins to you and you helping it take off.

What should I do next?

Mes­sage me here on LessWrong (or by other chan­nels if you know me), in­clud­ing the ma­te­rial you think is needed for me to eval­u­ate your fit, by June 10th. Don’t send me more than 2 para­graphs of text, al­though links to google docs for me to skim are fine. Please in­clude:

  • A brief sum­mary of past rele­vant ex­pe­rience of high-paced work

  • Some­thing show­ing your abil­ity to rea­son sen­si­bly about covid, or oth­er­wise rea­son un­der un­cer­tainty about real-life situ­a­tions. I can come to trust your judg­ment here by see­ing a his­tory of ex­cel­lent LessWrong posts.

  • Refer­ences (but only in case they are peo­ple I trust, or some­one I trust trusts).

If the main rea­son you don’t ex­pect to do this is be­cause you’re already quite busy, es­pe­cially if it’s at a non-profit, I might still be quite in­ter­ested in talk­ing through this op­por­tu­nity with you for half an hour so you have a clearer vi­sion of what op­por­tu­nity you’re not tak­ing. Your choice might be right but it’s of­ten good to prop­erly con­sider al­ter­na­tive hy­pothe­ses for an hour or two.

[Note: due to me be­ing very busy, Ben Pace wrote the ma­jor­ity of this post based on our con­ver­sa­tions. I en­dorse ev­ery­thing said, but might have for­mu­lated some things differ­ently had I writ­ten it my­self.]