I Vouch For MIRI

Another take with more links: AI: A Reason to Worry, A Reason to Donate

I have made a $10,000 dona­tion to the Machine In­tel­li­gence Re­search In­sti­tute (MIRI) as part of their winter fun­draiser. This is the best or­gan­iz­a­tion I know of to donate money to, by a wide mar­gin, and I en­cour­age oth­ers to also donate. This be­lief comes from a com­bin­a­tion of pub­lic in­form­a­tion, private in­form­a­tion and my own ana­lysis. This post will share some of my private in­form­a­tion and ana­lysis to help oth­ers make the best de­cisions.

I con­sider AI Safety the most im­port­ant, ur­gent and un­der-fun­ded cause. If your private in­form­a­tion and ana­lysis says an­other AI Safety or­gan­iz­a­tion is a bet­ter place to give, give to there. I be­lieve many AI Safety or­gan­iz­a­tions do good work. If you have the tal­ent and skills, and can get in­volved dir­ectly, or get oth­ers who have the tal­ent and skills in­volved dir­ectly, that’s even bet­ter than donat­ing money.

If you do not know about AI Safety and un­friendly ar­ti­fi­cial gen­eral in­tel­li­gence, I en­cour­age you to read about them. If you’re up for a book, read this one.

If you de­cide you care about other causes more, donate to those causes in­stead, in the way your ana­lysis says is most ef­fect­ive. Think for your­self, do and share your own ana­lysis, and con­trib­ute as dir­ectly as pos­sible.


I am very con­fid­ent in the fol­low­ing facts about ar­ti­fi­cial gen­eral in­tel­li­gence. None of my con­clu­sions in this sec­tion re­quire my private in­form­a­tion.

Hu­man­ity is likely to de­velop ar­ti­fi­cial gen­eral in­tel­li­gence (AGI) vastly smarter and more power­ful than hu­mans. We are un­likely to know that far in ad­vance when this is about to hap­pen. There is wide dis­agree­ment and un­cer­tainty on how long this will take, but cer­tainly there is sub­stan­tial chance this hap­pens within our life­times.

Whatever your pre­vi­ous be­liefs, the events of the last year, in­clud­ing AlphaGo Zero, should con­vince you that AGI is more likely to hap­pen, and more likely to hap­pen soon.

If we do build an AGI, its ac­tions will de­term­ine what is done with the uni­verse.

If the first such AGI we build turns out to be an un­friendly AI that is op­tim­iz­ing for some­thing other than hu­mans and hu­man val­ues, all value in the uni­verse will be des­troyed. We are made of atoms that could be used for some­thing else.

If the first such AGI we build turns out to care about hu­mans and hu­man val­ues, the uni­verse will be a place of value many or­ders of mag­nitude greater than it is now.

Al­most all AGIs that could be con­struc­ted care about some­thing other than hu­mans and hu­man val­ues, and would cre­ate a uni­verse with zero value. Mind­space is deep and wide, and al­most all of it does not care about us.

The de­fault out­come, if we do not work hard and care­fully now on AGI safety, is for AGI to wipe out all value in the uni­verse.

AI Safety is a hard prob­lem on many levels. Solv­ing it is much harder than it looks even with the best of in­ten­tions, and in­cent­ives are likely to con­spire to give those in­volved very bad per­sonal in­cent­ives. Without se­cur­ity mind­set, value align­ment and tons of ad­vance work, chances of suc­cess are very low.

We are cur­rently spend­ing ludicrously little time, at­ten­tion and money on this prob­lem.

For space reas­ons I am not fur­ther jus­ti­fy­ing these claims here. Ja­cob’s post has more links.


In these next two sec­tions I will share what I can of my own private in­form­a­tion and ana­lysis.

I know many prin­ciples at MIRI, in­clud­ing senior re­search fel­low Eliezer Yudkowsky and ex­ec­ut­ive dir­ector Nate Sores. They are bril­liant, and are as ded­ic­ated as one can be to the cause of AI Safety and en­sur­ing a good fu­ture for the uni­verse. I trust them, based on per­sonal ex­per­i­ence with them, to do what they be­lieve is best to achieve these goals.

I be­lieve they have already done much ex­cep­tional and valu­able work. I have also read many of their re­cent pa­pers and found them ex­cel­lent.

MIRI has been in­valu­able in lay­ing the ground­work for this field. This is true both on the level of the field ex­ist­ing at all, and also on the level of think­ing in ways that might ac­tu­ally work.

Even today, most who talk about AI Safety sug­gest strategies that have es­sen­tially no chance of suc­cess, but at least they are talk­ing about it at all. MIRI is a large part of why they’re talk­ing at all. I be­lieve that some­thing as simple as these Deep­Mind AI Safety test en­vir­on­ments is good, help­ing re­search­ers un­der­stand there is a prob­lem much more deadly than al­gorithmic dis­crim­in­a­tion. The risk is that re­search­ers will real­ize a prob­lem ex­ists, then think ‘I’ve solved these prob­lems, so I’ve done the AI Safety thing’ when we need the ac­tual thing the most.

From the be­gin­ning, MIRI un­der­stood the AI Safety prob­lem is hard, re­quir­ing dif­fi­cult high-pre­ci­sion think­ing, and long term de­vel­op­ment of new ideas and tools. MIRI con­tin­ues to fight to turn con­cern about ‘AI Safety’ into con­cern about AI Safety.

AI Safety is so hard to un­der­stand that Eliezer Yudkowsky de­cided he needed to teach the world the art of ra­tion­al­ity so we could then un­der­stand AI Safety. He did ex­actly that, which is why this blog ex­ists.

MIRI is de­vel­op­ing tech­niques to make AGIs we can un­der­stand and pre­dict and prove things about. MIRI seeks to un­der­stand how agents can and should think. If AGI comes from such mod­els, this is a huge boost to our chances of suc­cess. MIRI is also work­ing on tech­niques to make ma­chine learn­ing based agents safer, in case that path leads to AGI first. Both tasks are valu­able, but I am es­pe­cially ex­cited by MIRI’s work on lo­gic.


Eliezer’s model was that if we teach people to think, then they can think about AI.

What I’ve come to real­ize is that when we try to think about AI, we also learn how to think in gen­eral.

The pa­per that con­vinced OpenPhil to in­crease its grant to MIRI was about Lo­gical In­duc­tion. That pa­per was im­press­ive and worth un­der­stand­ing, but even more im­press­ive and valu­able in my eyes is MIRI’s work on Func­tional De­cision The­ory. This is vi­tal to cre­at­ing an AGI that makes de­cisions, and has been in­valu­able to me as a hu­man mak­ing de­cisions. It gave me a much bet­ter way to un­der­stand, work with and ex­plain how to think about mak­ing de­cisions.

Our so­ci­ety be­lieves in and praises Causal De­cision The­ory, dis­miss­ing other con­sid­er­a­tions as ir­ra­tional. This has been a dis­aster on a level hard to com­pre­hend. It des­troys the found­a­tions of civil­iz­a­tion. If we could spread prac­tical, hu­man use of Func­tional De­cision The­ory, and de­bate on that basis, we could get out of much of our cur­rent mess. Thanks to MIRI, we have a strong formal state­ment of Func­tional De­cision The­ory.

Whenever I think about AI or AI Safety, read AI pa­pers or try to design AI sys­tems, I learn how to think as a hu­man. As a side ef­fect of MIRI’s work, my think­ing, and es­pe­cially my abil­ity to form­al­ize, ex­plain and share my think­ing, has been greatly ad­vanced. Their work even this year has been a great help.

MIRI does ba­sic re­search into how to think. We should ex­pect such re­search to con­tinue to pay large and un­ex­pec­ted di­vidends, even ig­nor­ing its im­pact on AI Safety.


I be­lieve it is al­ways im­port­ant to use strategies that are co­oper­at­ive and in­form­a­tion cre­at­ing, rather than de­fect­ing and in­form­a­tion des­troy­ing, and that pre­serve good in­cent­ives for all in­volved. If we’re not us­ing a de­cision al­gorithm that cares more about such con­sid­er­a­tions than max­im­iz­ing rev­enue raised, even when rais­ing for a cause as good as ‘not des­troy­ing all value in the uni­verse,’ it will not end well.

This means that I need to do three things. I need to share my in­form­a­tion, as best I can. I need to in­clude my own bi­ases, so oth­ers can de­cide whether and how much to ad­just for them. And I need to avoid us­ing strategies that would be dis­tort or mis­lead.

I have not been able to share all my in­form­a­tion above, due to a com­bin­a­tion of space, com­plex­ity and con­fid­en­ti­al­ity con­sid­er­a­tions. I have done what I can. Bey­ond that, I will simply say that what re­main­ing private in­form­a­tion I have on net points in the dir­ec­tion of MIRI be­ing a bet­ter place to donate money.

My own bi­ases here are clear. The ma­jor­ity of my friends come from the ra­tion­al­ity com­munity, which would not ex­ist ex­cept for Eliezer Yudkowsky. I met my wife Laura at a com­munity meetup. I know sev­eral MIRI mem­bers per­son­ally, con­sider them friends, and even ran a strategy meet­ing for them sev­eral years back at their re­quest. It would not be sur­pris­ing if such con­sid­er­a­tions in­flu­enced my judg­ment some­what. Such con­cerns go hand in hand with be­ing in a po­s­i­tion to do ex­tens­ive ana­lysis and ac­quire private in­form­a­tion. This is all the more reason to do your own think­ing and ana­lysis of these is­sues.

To avoid dis­tor­tions, I am giv­ing the money dir­ectly, without qual­i­fic­a­tions or gim­micks or match­ing funds. My hope is that this will be a costly sig­nal that I have thought long and hard about such ques­tions, and reached the con­clu­sion that MIRI is an ex­cel­lent place to donate money. OpenPhil has a prin­ciple that they will not fund more than half of any or­gan­iz­a­tion’s budget. I think this is an ex­cel­lent prin­ciple. There is more than enough money in the ef­fect­ive al­tru­ist com­munity to fully fund MIRI and other such worthy causes, but these funds rep­res­ent a great tempta­tion. They risk caus­ing great dis­tor­tions, and ty­ing up ac­tion with polit­ical con­sid­er­a­tions, des­pite every­one’s best in­ten­tions.

As small givers (at least, re­l­at­ive to some) our biggest value lies not in the use of the money it­self, but in the in­form­a­tion value of the costly sig­nal our dona­tions give and in the vir­tues we cul­tiv­ate in ourselves by giv­ing. I be­lieve MIRI can ef­fi­ciently util­ize far more money than it cur­rently has, but more than that this is me say­ing that I know them, I know their work, and I be­lieve in and trust them. I vouch for MIRI.