Opportunities for individual donors in AI safety


Over the past few years the availa­bil­ity of fund­ing for AI safety pro­jects has in­creased sig­nifi­cantly. Yet op­por­tu­ni­ties for in­di­vi­d­ual donors to find high im­pact grant op­por­tu­ni­ties re­main. In this post I re­view the re­cent his­tory of AI safety or­ga­ni­za­tions, spec­u­late on the ways in which fund­ing early AI safety or­ga­ni­za­tions may have im­pacted to­day’s AI safety land­scape, and on this ba­sis pro­pose some heuris­tics for find­ing un­tapped fund­ing op­por­tu­ni­ties:

  • Fo­cus on which peo­ple you’re bring­ing into the community

  • Per­sonal brands and plat­forms mat­ter, and have ac­counted for some of the largest suc­cesses of the community

  • Cul­ture and in­tel­lec­tual en­vi­ron­ments matter

  • Small amounts of fund­ing to­day may move much larger amounts of fund­ing in the future

(Cross-posted from EA fo­rum upon re­quest.)

Up­dated 3/​31: added info on Sin­gu­lar­ity Sum­mit and Max Teg­mark.


For early AI safety or­ga­ni­za­tions, fund­ing was a ma­jor bot­tle­neck. Dur­ing the 2000-2010 pe­riod, large donors were sparse and, with few ex­cep­tions, not mo­ti­vated to com­mit large amounts of cap­i­tal to such es­o­teric causes[1]. Small donors ac­counted for a sub­stan­tial frac­tion of the bud­get of or­ga­ni­za­tions such as SIAI[2], while oth­ers lo­cated them­selves within uni­ver­si­ties. A pri­vate donor look­ing for fund­ing op­por­tu­ni­ties at this time would quickly find mul­ti­ple or­ga­ni­za­tions with sig­nifi­cant room for fund­ing, and much of the work in mak­ing grants con­sisted in com­par­ing differ­ent fund­ing op­por­tu­ni­ties for coun­ter­fac­tual im­pact.

Over the past few years that situ­a­tion has changed. Mul­ti­ple large grant-mak­ing or­ga­ni­za­tions have iden­ti­fied AI safety as an im­por­tant cause area,[3] in­clud­ing the Open Philan­thropy Pro­ject[4], the Fu­ture of Life In­sti­tute[5], the Berkeley Ex­is­ten­tial Risk In­sti­tute[6], and the EA Long Term Fu­ture Fund[7]. While the amounts that each or­ga­ni­za­tion or in­di­vi­d­ual hopes to de­ploy within AI safety is not pub­lic (or in many cases not yet de­cided), it seems likely that the amount of fund­ing that would be de­ployed if there were many large, promis­ing fund­ing op­por­tu­ni­ties ex­ceeds the room for more fund­ing across cur­rently ex­ist­ing AI safety or­ga­ni­za­tions.

This changes the na­ture of fund­ing de­ci­sions, since in a world with more fund­ing than pro­jects, the de­ci­sion to fund a pro­ject looks more like an eval­u­a­tion of whether the pro­ject crosses some use­ful­ness thresh­old, and less like a com­par­i­son be­tween pro­jects.

In this post I pro­pose some ini­tial thoughts on how grant-mak­ers might ap­proach this changed fund­ing land­scape. Of par­tic­u­lar rele­vance to me is op­por­tu­ni­ties available to in­di­vi­d­ual donors act­ing in the midst of mul­ti­ple large foun­da­tions de­ploy­ing sig­nifi­cant amounts of cap­i­tal.

Every­thing here should be should be in­ter­preted as early thoughts on the topic, and I ex­pect to up­date sig­nifi­cantly based on feed­back from the com­mu­nity.


If to­day’s AI safety com­mu­nity is to have a mean­ingful effect on the over­all de­vel­op­ment of AI then I be­lieve it must at some point in the fu­ture grow sig­nifi­cantly be­yond its cur­rent size. By “grow” I mean an in­crease in ei­ther the num­ber of peo­ple in­volved, or a spread­ing of ideas, cul­ture, or pub­lic in­ter­est. The rea­son I be­lieve this is as fol­lows. There ap­pears at pre­sent to be a vast amount of tech­ni­cal work to do in AI safety, and an even greater amount of op­er­a­tional and en­g­ineer­ing work must surely fol­low. In the long term, it seems to me that for AI safety to suc­ceed, the to­tal amount of work be­ing done on AI safety in a given year must even­tu­ally be­come non-triv­ial com­pared to the to­tal amount of work be­ing done on AI. Yet at the mo­ment these two fields are sev­eral or­ders of mag­ni­tude apart in terms of peo­ple and out­put. Since the AI field it­self is cur­rently grow­ing quickly, it seems un­likely that the AI safety com­mu­nity will have a mean­ingful im­pact on the fu­ture de­vel­op­ment of AI if it never grows be­yond its cur­rent size.

This does not im­ply that the com­mu­nity should grow im­me­di­ately or that it needs to grow rapidly, only that it must even­tu­ally grow if it is to have a mean­ingful im­pact. Thank­fully, the com­mu­nity looks to me to be well poised for sig­nifi­cant growth over the com­ing years and decades. When mak­ing fund­ing de­ci­sions, grant mak­ers should there­fore think care­fully about how fund­ing de­ci­sions flow through sce­nar­ios in which there is sig­nifi­cant growth at some point in the fu­ture.

If the com­mu­nity does grow sig­nifi­cantly, then much of our raw out­put (re­search, out­reach, policy pro­pos­als, and so on) will be in the fu­ture, where there are larger num­bers of peo­ple do­ing di­rect work, and those peo­ple are bet­ter in­formed by what has or has not worked so far, have more in­for­ma­tion on how the over­all AI land­scape is evolv­ing, and so on. For this rea­son we may have sub­stan­tial lev­er­age now, since small efforts to­day may af­fect the work done by the larger fu­ture AI safety com­mu­nity: what its fo­cus is, how clear its re­search agenda is, who is in­volved, and how it is per­ceived ex­ter­nally.

I there­fore think that much of the im­pact of fund­ing de­ci­sions to­day flow through this larger fu­ture AI safety com­mu­nity: that is, ei­ther the com­mu­nity does not grow, and grants made to­day have lit­tle im­pact on the long term fu­ture of AI, or else the com­mu­nity does grow, and the im­pact of to­day’s grants play out via their effects on the fu­ture where most of the to­tal work is done.

Ex­am­ples from re­cent history

If the im­pact of to­day’s fund­ing de­ci­sions mostly flow through to­mor­row’s AI safety com­mu­nity, then it may be helpful to un­der­stand how fund­ing de­ci­sions made in the past have af­fected to­day’s AI safety com­mu­nity. In this sec­tion I will give a brief run­down of the old­est or­ga­ni­za­tions in the space, and some thoughts on how their ac­tions have shaped to­day’s AI safety com­mu­nity.

Un­for­tu­nately the task of re­con­struct­ing a de­tailed his­tory of these or­ga­ni­za­tions is a much larger pro­ject, so I’m go­ing to pre­sent here some very rough notes based on pub­lished timelines[8] and on my own un­der­stand­ing of how these or­ga­ni­za­tions evolved. If my re­con­struc­tion here is in­cor­rect then I hope that this lens will still provide an in­tu­ition pump for think­ing about how fund­ing de­ci­sions made to­day may af­fect the fu­ture.

Some caveats:

  • I am ar­bi­trar­ily in­clud­ing ev­ery­thing up to 2010 as “re­cent his­tory”.

  • I am try­ing to as­sess the ways in which the early forms of to­day’s or­ga­ni­za­tions af­fected the pre­sent land­scape.

  • I am not try­ing to as­sign pos­i­tive or nega­tive value to these effects.

  • As always, it is very difficult to prop­erly al­lo­cate coun­ter­fac­tual credit and this is just a first pass.

Sin­gu­lar­ity In­sti­tute for Ar­tifi­cial Intelligence

SIAI was founded by Eliezer Yud­kowsky in July 2000[9] with the stated mis­sion of “cre­at­ing a friendly, self-im­prov­ing ar­tifi­cial in­tel­li­gence”. In 2013 the or­ga­ni­za­tion was re­named to the Ma­chine In­tel­li­gence Re­search In­sti­tute. Here I list some of the work that SIAI en­gaged in dur­ing these years and my best guess as to their effects on to­day’s AI safety com­mu­nity.

  • Direct re­search

    • It seems that some of the early Yud­kowsky pub­li­ca­tions (pre-2012) con­vinced key in­di­vi­d­u­als to take AI safety se­ri­ously.

    • But to­day’s AI safety liter­a­ture does not seem to build di­rectly on con­cep­tual foun­da­tions laid by this early work.

    • The main ex­cep­tion ap­pears to be the work on de­ci­sion the­ory, which does seem to trace its roots di­rectly to tech­ni­cal work from this pe­riod.

  • Less­wrong

    • Ap­pears to have caused a sub­stan­tial num­ber of peo­ple to de­vote their ca­reers to work­ing on AI safety, and many more to be­come in­volved out­side of full time work.

    • The spe­cific ideas writ­ten about in the se­quences are clearly widely known, but it is un­clear to me how much im­pact they have had on to­day’s tech­ni­cal AI safety land­scape.

    • The cul­ture cre­ated within less­wrong ap­pears to have spread via nu­mer­ous less­wrong mee­tups. My sense is that this cul­ture has had sig­nifi­cant effects on the cul­ture of to­day’s AI safety com­mu­nity.

  • Visit­ing Fel­lows Pro­gram

    • I per­son­ally know of two in­di­vi­d­u­als who trace their full time in­volve­ment with AI safety to the SI sum­mer fel­lows pro­gram.

    • On this ba­sis I es­ti­mate that there are at least two more in­di­vi­d­u­als that I am not aware of.

  • Sin­gu­lar­ity Sum­mit

  • Public out­reach

    • What kinds of PR did Eliezer do in the very early days? What­ever he did, it does not seem to have sig­nifi­cantly shaped to­day’s pub­lic per­cep­tion of AI or AI safety.

Over­all, Sin­gu­lar­ity In­sti­tute may de­serve par­tial credit for the very ex­is­tence of an AI safety com­mu­nity to­day. My very rough sense is that much of its im­pact ap­pears to have flowed through cul­ture cre­ation and bring­ing peo­ple into the com­mu­nity. Its early pub­li­ca­tions ap­pear to have had less di­rect im­pact on shap­ing the re­search agen­das be­ing pur­sued to­day.

Fu­ture of Hu­man­ity Institute

FHI was es­tab­lished by Nick Bostrom and An­ders Sand­berg in 2005 with fund­ing from James Martin and the Bright Hori­zons Foun­da­tion. [10] Some of the ac­tivi­ties that FHI un­der­took in the years up to 2012 were:

  • Fund­ing of the blog Over­com­ing Bias

  • Or­ga­ni­za­tion of Whole Brain Emu­la­tion Workshop

  • Direct re­search

    • Pub­lish­ing of Global Catas­trophic Risks

    • Pub­lish­ing of An­thropic Bias

    • Pub­lish­ing of Hu­man Enhancement

    • Numer­ous pa­pers by Bostrom, Sand­berg, and others

  • Creation of an in­tel­lec­tual micro­cosm

    • Toby Ord was a re­searcher at FHI when he and oth­ers launched Giv­ing What We Can, which led di­rectly to the for­ma­tion of the Effec­tive Altru­ism move­ment.

    • How­ever, Toby has in­di­cated that he does not think the ex­is­tence of FHI had a mean­ingful im­pact on his de­ci­sion to found GWWC.

FHI ap­pears to have placed less em­pha­sis on AI safety in its early days com­pared to the pre­sent day. To my mind, it looks as if much of the im­pact of FHI’s early work flows through the later pub­lish­ing of Su­per­in­tel­li­gence by Bostrom. To a first ap­prox­i­ma­tion, the early FHI re­search ap­pears to have had much of its im­pact by giv­ing Bostrom the cred­i­bil­ity and plat­form that al­lowed his later book to have such a broad re­cep­tion.

One might also spec­u­late about what if any con­tri­bu­tion FHI’s early work had in spurring Toby Ord to start Giv­ing What We Can by Toby Ord in 2009, which was one of the foun­da­tions on which the early Effec­tive Altru­ism com­mu­nity was built. Ord and Bostrom were pub­lish­ing to­gether as early as 2006[11]. On the other hand, there are no other FHI staff on the list of early Giv­ing What We Can pledges[12], and Toby him­self does not think FHI had a mean­ingful im­pact on the cre­ation of GWWC.


GiveWell was founded in 2008 by Holden Karnofsky and Elie Hassen­feld to study philan­thropic giv­ing op­por­tu­ni­ties and make recom­men­da­tions to small and medium-sized donors. GiveWell was never fo­cused on AI safety, al­though GiveWell Labs did con­duct an in­for­mal eval­u­a­tion of SIAI in 2012, which was nega­tive.

From an AI safety per­spec­tive, GiveWell’s largest im­pact has al­most cer­tainly been the cre­ation of the Open Philan­thropy Pro­ject, which is the pri­mary ad­viser to the multi-billion dol­lar fund, Good Ven­tures, and has iden­ti­fied risks from ad­vanced AI as a fo­cus area.

It seems to me that the largest im­pacts from GiveWell’s early work flowed through the peo­ple who went on to cre­ate OpenPhil, and the cul­ture of ap­proach­ing philan­thropy as an in­tel­lec­tu­ally se­ri­ous en­deavor. The di­rect work performed by GiveWell was al­most cer­tainly cru­cial in at­tract­ing tal­ent to the cause, and cap­tur­ing the at­ten­tion of Good Ven­tures, but this im­pact ap­pears to flow to a large ex­tent through peo­ple and cul­ture, both of which were crit­i­cal to the later cre­ation of Ope­hPhil.

Heuris­tics for fund­ing AI safety organizations

Fo­cus on People

Some of the largest im­pacts of fund­ing FHI or SIAI in their early years ap­pear to have flown through the in­di­vi­d­u­als those or­ga­ni­za­tions hired, who went on to con­tribute sub­stan­tially within the com­mu­nity. I sus­pect that the early re­search performed by these or­ga­ni­za­tions was im­pact­ful pri­mar­ily in­so­far as it at­tracted cer­tain in­di­vi­d­u­als into the com­mu­nity.

One lens through which grant mak­ers may view fund­ing de­ci­sions is there­fore as a sort of hiring de­ci­sion in which the grant maker asks which peo­ple will be moved into the AI safety com­mu­nity as a re­sult of the grant. For or­ga­ni­za­tions that would not oth­er­wise re­ceive fund­ing, this may en­com­pass all in­di­vi­d­u­als in the or­ga­ni­za­tion. For oth­ers, the grant-maker is mov­ing whichever in­di­vi­d­u­als end up be­ing hired as a re­sult of the grant.

When mak­ing a hiring de­ci­sion, it is of course im­por­tant to re­view past work and eval­u­ate the im­pact that the can­di­date’s planned work will have, yet most hiring de­ci­sions also place sub­stan­tial weight on a more neb­u­lous sense of the po­ten­tial of bring­ing in­tel­li­gent, mo­ti­vated, value-al­igned in­di­vi­d­u­als into an or­ga­ni­za­tion. Grant mak­ers should similarly take this more neb­u­lous com­po­nent se­ri­ously, since it ap­pears to have ac­counted for a sig­nifi­cant amount of the im­pact of past fund­ing de­ci­sions in the field.

In­ter­est­ingly, this matches the oft-quoted wis­dom among startup in­vestors of mak­ing early-stage in­vest­ments largely on the strength of the team that runs the com­pany.

Plat­forms Matter

Bostrom’s book Su­per­in­tel­li­gence prob­a­bly could not have gained such wide­spread ex­po­sure with­out Bostrom hav­ing built up a se­ri­ous aca­demic plat­form[13]. The plat­form on which he launched Su­per­in­tel­li­gence was built over 14 years, and dur­ing this time he au­thored more than a hun­dred pa­pers, pre­sented at many hun­dreds of con­fer­ences and work­shops, and gave nu­mer­ous me­dia ap­pear­ances.[14] Yet a large com­po­nent of im­pact of all this work thus far does not seem to have been the di­rect in­sights gleaned or the di­rect effects of pub­lic out­reach on spe­cific is­sues, but in­stead seems to have flown through the cre­ation of a plat­form upon which his later views on AI safety could reach a broad au­di­ence.

A grant maker con­sid­er­ing fund­ing FHI dur­ing the 2000-2010 pe­riod may have been tempted to eval­u­ate the im­pact of the di­rect work that the or­ga­ni­za­tion planned to pur­sue in the short term, but this would have largely missed the most im­por­tant re­source be­ing gen­er­ated at FHI.

Cul­ture and In­tel­lec­tual Environments

Through the cre­ation of LessWrong, SIAI caused a wor­ld­wide net­work to co­a­lesce around ideas in philos­o­phy, math, and AI. This net­work had dis­tinc­tive cul­ture that ap­pears to me to have had a sub­stan­tial im­pact on to­day’s AI safety com­mu­nity[15]. The long-term cul­tural im­pacts of FHI ap­pears to have been even larger, par­tic­u­larly if we give it par­tial credit for cre­at­ing the in­tel­lec­tual en­vi­ron­ment in which Giv­ing What We Can and even­tu­ally the broader Effec­tive Altru­ism move­ment formed.

Cul­ture is difficult to mea­sure or pre­dict. Early fun­ders of SIAI or FHI would have had difficulty fore­see­ing the long-term cul­tural im­pacts these or­ga­ni­za­tions would have, yet these cul­tural out­comes do ap­pear to have been a very sig­nifi­cant com­po­nent of the re­spec­tive or­ga­ni­za­tions’ im­pact, so are worth tak­ing se­ri­ously.

This also matches a wide­spread be­lief among tech­nol­ogy star­tups that early com­pany cul­ture is a crit­i­cal de­ter­mi­nant of long term suc­cess or failure.

Fund­ing begets fur­ther funding

If the growth the­sis above is cor­rect then mak­ing grants to­day has the po­ten­tial to move many times as much money in the fu­ture, if mak­ing a grant now causes an or­ga­ni­za­tion to be more likely to ex­ist, or be more promi­nent dur­ing a fu­ture AI safety growth phase.

Early GiveWell sup­port­ers surely de­serve some credit for the much larger amounts of money that OpenPhil is now mov­ing. Similarly, early FHI and SIAI back­ers de­serve some credit for the more sub­stan­tial pre­sent day bud­gets of those or­ga­ni­za­tions.

For AI safety grant mak­ers to­day, this im­plies that fund­ing early-stage or­ga­ni­za­tions may have a par­tic­u­larly large im­pact, since it can make the differ­ence be­tween an or­ga­ni­za­tion ex­ist­ing and not ex­ist­ing dur­ing a fu­ture growth phase. This pro­vides one ar­gu­ment in fa­vor of giv­ing now rather than giv­ing later.[16]

Feed­back Loops

It looks to me that the largest effects of fund­ing AI safety or­ga­ni­za­tions will ma­te­ri­al­ize through growth and the feed­back loops that drive them:

  • Cul­tural feed­back

    • Ini­tial effect: found­ing team lays down a cer­tain cul­ture within an organization

    • Feed­back effect: as new peo­ple en­ter the or­ga­ni­za­tion they adopt the cul­ture of those already in the organization

  • Hiring feed­back

    • Ini­tial effect: an or­ga­ni­za­tion uses money from a grant to hire people

    • Feed­back effect: some of those peo­ple start new or­ga­ni­za­tions, which con­tinue to hire more peo­ple, or con­tribute to hiring decisions

  • Memetic feed­back

    • Ini­tial effect: by en­coun­ter­ing work pub­lished by an or­ga­ni­za­tion, in­di­vi­d­u­als out­side the AI safety com­mu­nity ad­just their views on AI safety

    • Feed­back effect: memes prop­a­gate to fur­ther individuals

  • Fund­ing feed­back

    • Ini­tial effect: an or­ga­ni­za­tion re­ceives a grant

    • Feed­back effect: other grant mak­ers choose whether to fund­ing this and similar or­ga­ni­za­tions based on its per­ceived suc­cess or failure


With the en­try of sev­eral large grant mak­ers to the AI safety space, many of the largest or­ga­ni­za­tions ap­pear to have ex­hausted their ca­pac­ity for ad­di­tional fund­ing. Fur­ther­more, the bud­gets of these or­ga­ni­za­tions re­main small com­pared to the to­tal cap­i­tal that AI safety grant mak­ers hope to de­ploy in the next few years, so the re-emer­gence of sig­nifi­cant fund­ing gaps among well-es­tab­lished or­ga­ni­za­tions seems un­likely even as these or­ga­ni­za­tions scale up.

Nev­er­the­less, in­di­vi­d­ual grant mak­ers may hope to find large op­por­tu­ni­ties for im­pact by iden­ti­fy­ing fund­ing op­por­tu­ni­ties among small or­ga­ni­za­tions, since we must as­sume that the AI safety com­mu­nity will grow sub­stan­tially at some point if it is not have sub­stan­tial im­pact.

When mak­ing fund­ing de­ci­sions, grant mak­ers should pay spe­cial at­ten­tion to op­por­tu­ni­ties to “hire” peo­ple into the over­all AI safety com­mu­nity, and the unique cul­ture that some or­ga­ni­za­tions foster. In ad­di­tion, grant mak­ers should seize op­por­tu­ni­ties to help in­di­vi­d­u­als build up plat­forms upon which their ideas can reach a wide au­di­ence, and be aware that grants made to­day may move much larger sums of money in the fu­ture.

Donors who take these neb­u­lous fac­tors se­ri­ously may find op­por­tu­ni­ties for im­pact missed by grant mak­ers that fo­cus more on the im­me­di­ate work gen­er­ated by grants.


[1] For one snap­shot of the fund­ing land­scape from 2003 un­til 2010, see http://​less­wrong.com/​lw/​5il/​siai_an_ex­am­i­na­tion/​

[2] The Sin­gu­lar­ity In­sti­tute for Ar­tifi­cial In­tel­li­gence, which was re­named to the Ma­chine In­tel­li­gence Re­search In­sti­tute in 2014

[3] See e.g. https://​www.open­philan­thropy.org/​fo­cus/​global-catas­trophic-risks/​po­ten­tial-risks-ad­vanced-ar­tifi­cial-intelligence

[4] https://​www.open­philan­thropy.org/​fo­cus/​global-catas­trophic-risks/​po­ten­tial-risks-ad­vanced-ar­tifi­cial-in­tel­li­gence/​

[5] https://​fu­ture­oflife.org/​back­ground/​benefits-risks-of-ar­tifi­cial-in­tel­li­gence/​

[6] http://​ex­is­tence.org/​grants

[7] https://​app.effec­tivealtru­ism.org/​funds/​far-future

[8] E.g. https://​timelines.is­sarice.com/​wiki/​Timeline_of_Ma­chine_In­tel­li­gence_Re­search_Institute

[9] https://​timelines.is­sarice.com/​wiki/​Timeline_of_Ma­chine_In­tel­li­gence_Re­search_Institute

[10] https://​web.archive.org/​web/​20060512085807/​http://​www.fhi.ox.ac.uk:80/​Papers/​FHI%20Newslet­ter%201%20-%20April%20200611.pdf

[11] Nick Bostrom, and Toby Ord. “The re­ver­sal test: elimi­nat­ing sta­tus quo bias in ap­plied ethics.“Ethics 116.4 (2006): 656-679.)

[12] https://​www.giv­ing­whatwe­can.org/​about-us/​mem­bers/​

[13] Look­ing through the non­fic­tion cat­e­gory of the New York Times best­sel­lers list, I see few pub­lished by in­di­vi­d­u­als who do not already have a sub­stan­tial plat­form.

[14] https://​nick­bostrom.com/​cv.pdf

[15] This is very difficult to jus­tify ex­plic­itly. All I can offer is my own in­tu­itions.

[16] Just one ar­gu­ment of many; for a more com­plete treat­ment see https://​ra­tio­nalaltru­ist.com/​2013/​03/​12/​giv­ing-now-vs-later/​

[17] Per­sonal cor­re­spon­dence with Carl Shulman