Course recommendations for Friendliness researchers

When I first learned about Friendly AI, I as­sumed it was mostly a pro­gram­ming prob­lem. As it turns out, it’s ac­tu­ally mostly a math prob­lem. That’s be­cause most of the the­ory be­hind self-refer­ence, de­ci­sion the­ory, and gen­eral AI tech­niques haven’t been for­mal­ized and solved yet. Thus, when peo­ple ask me what they should study in or­der to work on Friendli­ness the­ory, I say “Go study math and the­o­ret­i­cal com­puter sci­ence.”

But that’s not spe­cific enough. Should as­piring Friendli­ness re­searchers study con­tin­u­ous or dis­crete math? Im­per­a­tive or func­tional pro­gram­ming? Topol­ogy? Lin­ear alge­bra? Ring the­ory?

I do, in fact, have spe­cific recom­men­da­tions for which sub­jects Friendli­ness re­searchers should study. And so I worked with a few of my best in­terns at MIRI to provide recom­men­da­tions be­low:

  • Univer­sity courses. We care­fully hand-picked courses on these sub­jects from four lead­ing uni­ver­si­ties — but we aren’t om­ni­scient! If you’re at one of these schools and can give us feed­back on the ex­act courses we’ve recom­mended, please do so.

  • On­line courses. We also linked to on­line courses, for the ma­jor­ity of you who aren’t able to at­tend one of the four uni­ver­si­ties whose course cat­a­logs we dug into. Feed­back on these on­line courses is also wel­come; we’ve only taken a few of them.

  • Text­books. We have read nearly all the text­books recom­mended be­low, along with many of their com­peti­tors. If you’re a strongly mo­ti­vated au­to­di­dact, you could learn these sub­jects by div­ing into the books on your own and do­ing the ex­er­cises.

Have you already taken most of the sub­jects be­low? If so, and you’re in­ter­ested in Friendli­ness re­search, then you should definitely con­tact me or our pro­ject man­ager Malo Bour­gon (malo@in­tel­li­ You might not feel all that spe­cial when you’re in a top-notch math pro­gram sur­rounded by peo­ple who are as smart or smarter than you are, but here’s the deal: we rarely get con­tacted by as­piring Friendli­ness re­searchers who are fa­mil­iar with most of the ma­te­rial be­low. If you are, then you are spe­cial and we want to talk to you.

Not ev­ery­one cares about Friendly AI, and not ev­ery­one who cares about Friendly AI should be a re­searcher. But if you do care and you might want to help with Friendli­ness re­search one day, we recom­mend you con­sume the sub­jects be­low. Please con­tact me or Malo if you need fur­ther guidance. Or when you’re ready to come work for us.

PHIL 190
Free Online
Cog­ni­tive Science
If you’re en­deav­or­ing to build a mind, why not start by study­ing your own? It turns out we know quite a bit: hu­man minds are mas­sively par­allel, highly re­dun­dant, and al­though parts of the cor­tex and neo­cor­tex seem re­mark­ably uniform, there are definitely dozens of spe­cial pur­pose mod­ules in there too. Know the ba­sic de­tails of how the only ex­ist­ing gen­eral pur­pose in­tel­li­gence cur­rently func­tions.
ECON 119
IPS 207A
Free Online
Heuris­tics and Biases
While cog­ni­tive sci­ence will tell you all the won­der­ful things we know about the im­mense, par­allel na­ture of the brain, there’s also the other side of the coin. Evolu­tion de­signed our brains to be op­ti­mized at do­ing rapid thought op­er­a­tions that work in 100 steps or less. Your brain is go­ing to make stuff up to cover up that its mostly cut­ting cor­ners. Th­ese er­rors don’t feel like er­rors from the in­side, so you’ll have to learn how to patch the ones you can and then move on.

PS—We should prob­a­bly de­sign our AIs bet­ter than this.
MATH 198
Free Online
Func­tional Programing
There are two ma­jor branches of pro­gram­ming: Func­tional and Im­per­a­tive. Un­for­tu­nately, most pro­gram­mers only learn im­per­a­tive pro­gram­ming lan­guages (like C++ or python). I say un­for­tu­nately, be­cause these lan­guages achieve all their power through what pro­gram­mers call “side effects”. The ma­jor down­side for us is that this means they can’t be effi­ciently ma­chine checked for safety or cor­rect­ness. The first self-mod­ify­ing AIs will hope­fully be writ­ten in func­tional pro­gram­ming lan­guages, so learn some­thing use­ful like Haskell or Scheme.
CME 305
Free Online
Discrete Math
Much like pro­gram­ming, there are two ma­jor branches of math­e­mat­ics as well: Discrete and con­tin­u­ous. It turns out a lot of physics and all of mod­ern com­pu­ta­tion is ac­tu­ally dis­crete. And al­though con­tin­u­ous ap­prox­i­ma­tions have oc­ca­sion­ally yielded use­ful re­sults, some­times you just need to calcu­late it the dis­crete way. Un­for­tu­nately, most en­g­ineers squan­der the ma­jor­ity of their aca­demic ca­reers study­ing higher and higher forms of calcu­lus and other con­tin­u­ous math­e­mat­ics. If you care about AI, study dis­crete math so you can un­der­stand com­pu­ta­tion and not just elec­tric­ity.

Also, you should pick up enough graph the­ory in this course to han­dle the ba­sic me­chan­ics of de­ci­sion the­ory—which you’re gonna want to learn later.
MATH 110
MATH 113
Free Online
Lin­ear Algebra
Lin­ear alge­bra is the foun­da­tion of quan­tum physics and a huge amount of prob­a­bil­ity the­ory. It even shows up in analy­ses of things like neu­ral net­works. You can’t pos­si­bly get by in ma­chine learn­ing (later) with­out speak­ing lin­ear alge­bra. So learn it early in your scholas­tic ca­reer.
MATH 135
MATH 161
Free Online
Set Theory
Like learn­ing how to read in math­e­mat­ics. But in­stead of build­ing up let­ters into words, you’ll be build­ing up ax­ioms into the­o­rems. This will in­tro­duce you to the pro­gram of us­ing ax­ioms to cap­ture in­tu­ition, find­ing prob­lems with the ax­ioms, and fix­ing them.
CS 103
Free Online
Math­e­mat­i­cal Logic
The math­e­mat­i­cal equiv­a­lent of build­ing words into sen­tences. Essen­tial for the math­e­mat­ics of self-mod­ifi­ca­tion. And even though Sher­lock Holmes and other pop­u­lar de­pic­tions make it look like magic, it’s just lawful for­mu­las all the way down.
CS 161
Free Online
Effi­cient Al­gorithms and In­tractable Problems
Like build­ing sen­tences into para­graphs. Al­gorithms are the recipes of thought. One of the more amaz­ing things about al­gorithm de­sign is that it’s of­ten pos­si­ble to tell how long a pro­cess will take to solve a prob­lem be­fore you ac­tu­ally run the pro­cess to check it. Learn­ing how to de­sign effi­cient al­gorithms like this will be a foun­da­tional skill for any­one pro­gram­ming an en­tire AI, since AIs will be built en­tirely out of col­lec­tions of al­gorithms.
Free Online
Numer­i­cal Analysis
There are ways to sys­tem­at­i­cally de­sign al­gorithms that only get things slightly wrong when the in­put data has tiny er­rors. And then there’s pro­grams writ­ten by am­a­teur pro­gram­mers who don’t take this class. Most pro­gram­mers will skip this course be­cause it’s not re­quired. But for us, get­ting the right an­swer is very much re­quired.
CS 154
Free Online
Com­putabil­ity and Complexity
This is where you get to study com­put­ing at it’s most the­o­ret­i­cal. Learn about the Church-Tur­ing the­sis, the uni­ver­sal na­ture and ap­pli­ca­bil­ity of com­pu­ta­tion, and how just like AIs, ev­ery­thing else is al­gorithms… all the way down.
CS 259Q
Free Online
Quan­tum Computing
It turns out that our uni­verse doesn’t run on Tur­ing Machines, but on quan­tum physics. And some­thing called BQP is the class of al­gorithms that are ac­tu­ally effi­ciently com­putable in our uni­verse. Study­ing the effi­ciency of al­gorithms rel­a­tive to clas­si­cal com­put­ers is use­ful if you’re pro­gram­ming some­thing that only needs to work to­day. But if you need to know what is effi­ciently com­putable in our uni­verse (at the limit) from a the­o­ret­i­cal per­spec­tive, quan­tum com­put­ing is the only way to un­der­stand that.
Free Online
Par­allel Computing
There’s a good chance that the first true AIs will have at least some al­gorithms that are in­effi­cient. So they’ll need as much pro­cess­ing power as we can throw at them. And there’s ev­ery rea­son to be­lieve that they’ll be run on par­allel ar­chi­tec­tures. There are a ton of is­sues that come up when you switch from as­sum­ing se­quen­tial in­struc­tion or­der­ing to par­allel pro­cess­ing. There’s thread­ing, dead­locks, mes­sage pass­ing, etc. The good part about this course is that most of the prob­lems are pinned down and solved: You’re just learn­ing the prac­tice of some­thing that you’ll need to use as a tool, but won’t need to ex­tend much (if any).
EE 219C
Free Online
Au­to­mated Pro­gram Verification
Re­mem­ber how I told you to learn func­tional pro­gram­ming way back at the be­gin­ning? Now that you wrote your code in func­tional style, you’ll be able to do au­to­mated and in­ter­ac­tive the­o­rem prov­ing on it to help ver­ify that your code matches your specs. Er­rors don’t make pro­grams bet­ter and all large pro­grams that aren’t for­mally ver­ified are re­li­ably *full* of er­rors. Ex­perts who have thought about the prob­lem for more than 5 min­utes agree that in­cor­rectly de­signed AI could cause dis­asters, so world-class cau­tion is ad­vis­able.
CS 109
Free Online
Com­bi­na­torics and Discrete Probability
Life is un­cer­tain and AIs will han­dle that un­cer­tainty us­ing prob­a­bil­ities. Also, prob­a­bil­ity is the foun­da­tion of the mod­ern con­cept of ra­tio­nal­ity and the mod­ern field of ma­chine learn­ing. Prob­a­bil­ity the­ory has the same foun­da­tional sta­tus in AI that logic has in math­e­mat­ics. Every­thing else is built on top of prob­a­bil­ity.
Free On­line
Bayesian Model­ing and Inference
Now that you’ve learned how to calcu­late prob­a­bil­ities, how do you com­bine and com­pare all the prob­a­bil­is­tic data you have? Like many choices be­fore, there is a dom­i­nant paradigm (fre­quen­tism) and a minor­ity paradigm (Bayesi­anism). If you learn the wrong method here, you’re de­vi­at­ing from a know­ably cor­rect frame­work for in­te­grat­ing de­grees of be­lief about new in­for­ma­tion and em­brac­ing a cadre of spe­cial pur­pose, ad-hoc statis­ti­cal solu­tions that of­ten break silently and with­out warn­ing. Also, quite em­bar­rass­ingly, fre­quen­tism’s abil­ity to get things right is bounded by how well it later turns out to have agreed with Bayesian meth­ods any­way. Why not just do the cor­rect thing from the be­gin­ning and not have your lunch eaten by Bayesi­ans ev­ery time you and them dis­agree?
MATH 218A/​B
MATH 230A/​B/​C
Free On­line
Prob­a­bil­ity Theory
No more ap­plied prob­a­bil­ity: Here be the­ory! Deep the­o­ries of prob­a­bil­ities are some­thing you’re go­ing to have to ex­tend to help build up the field of AI one day. So you ac­tu­ally have to know why all the things you’re do­ing are work­ing in­side out.
CS 229
Free Online
Ma­chine Learning
Now that you chose the right branch of math, the right kind of statis­tics, and the right pro­gram­ming paradigm, you’re pre­pared to study ma­chine learn­ing (aka statis­ti­cal learn­ing the­ory). There are lots of al­gorithms that lev­er­age prob­a­bil­is­tic in­fer­ence. Here you’ll start learn­ing tech­niques like clus­ter­ing, mix­ture mod­els, and other things that cache out as pre­cise, tech­ni­cal defi­ni­tions of con­cepts that nor­mally have rather con­fused or con­fus­ing English defi­ni­tions.
CS 221
Free Online
Ar­tifi­cial Intelligence
We made it! We’re fi­nally do­ing some AI work! Do­ing log­i­cal in­fer­ence, heuris­tic de­vel­op­ment, and other tech­niques will lev­er­age all the stuff you just learned in ma­chine learn­ing. While mod­ern, main­stream AI has many use­ful tech­niques to offer you, the au­thors will tell you out­right that, “the princess is in an­other cas­tle”. Or rather, there isn’t a princess of gen­eral AI al­gorithms any­where—not yet. We’re gonna have to go back to math­e­mat­ics and build our own meth­ods our­selves.
MATH 136
PHIL 152
Free Online
In­com­plete­ness and Undecidability
Prob­a­bly the most cel­e­brated re­sults is math­e­mat­ics are the nega­tive re­sults by Kurt Goedel: No finite set of ax­ioms can al­low all ar­ith­metic state­ments to be de­cided as ei­ther true or false… and no set of self-refer­en­tial ax­ioms can even “be­lieve” in its own con­sis­tency. Well, that’s a darn shame, be­cause re­cur­sively self-im­prov­ing AI is go­ing to need to side-step these the­o­rems. Even­tu­ally, some­one will un­lock the key to over-com­ing this difficulty with self-refer­ence, and if you want to help us do it, this course is part of the train­ing ground.
MATH 225A/​B
PHIL 151
Free On­line
Work­ing within a frame­work of math­e­mat­ics is great. Work­ing above math­e­mat­ics—on math­e­mat­ics—with math­e­mat­ics, is what this course is about. This would seem to be the most ob­vi­ous first step to over­com­ing in­com­plete­ness some­how. Prob­lem is, it’s definitely not the whole an­swer. But it would be sur­pris­ing if there were no clues here at all.
MATH 229
Free Online
Model Theory
One day, when some­one does side-step self-refer­ence prob­lems enough to pro­gram a re­cur­sively self-im­prov­ing AI, the guy sit­ting next to her who glances at the solu­tion will go “Gosh, that’s a nice bit of Model The­ory you got there!”

Think of Model The­ory as a for­mal way to un­der­stand what “true” means.
MATH 198
Free Online
Cat­e­gory Theory
Cat­e­gory the­ory is the pre­cise way that you check if struc­tures in one branch of math rep­re­sent the same struc­tures some­where else. It’s a re­mark­able field of meta-math­e­mat­ics that nearly no one knows… and it could hold the keys to im­port­ing use­ful tools to help solve dilem­mas in self-refer­ence, truth, and con­sis­tency.
Out­side recommendations
Harry Pot­ter and the Meth­ods of Rationality
Highly recom­mended book of light, en­joy­able read­ing that pre­dictably in­spires peo­ple to re­al­ize FAI is an im­por­tant prob­lem AND that they should prob­a­bly do some­thing about that.

You can start read­ing this im­me­di­ately, be­fore any of the above courses.
Global Catas­trophic Risks
A good primer on xrisks and why they might mat­ter. SPOILER ALERT: They mat­ter.

You can prob­a­bly skim read this early on in your stud­ies. Right af­ter HP:MoR.
The Sequences
Ra­tion­al­ity: the in­dis­pens­able art of non-self-de­struc­tion! There are man­i­fold ways you can fail at life… es­pe­cially since your brain is made out of bro­ken, un­doc­u­mented spaghetti code. You should learn more about this ASAP. That goes dou­ble if you want to build AIs.

I highly recom­mend you read this be­fore you get too deep into your aca­demic ca­reer. For in­stance, I know peo­ple who went to col­lege for 5 years, while some­how man­ag­ing to learn noth­ing. That’s be­cause in­stead of learn­ing, they merely re­cited the teacher’s pass­word ev­ery semester un­til they could dump what­ever they “learned” out of their heads as soon as they walked out of the fi­nal. Don’t let this hap­pen to you! This, and a hun­dred other use­ful les­sons like it about how to avoid pre­dictable, uni­ver­sal er­rors in hu­man rea­son­ing and be­hav­ior await you in The Se­quences!
Good and Real
A sur­pris­ingly thought­ful book on de­ci­sion the­ory and other para­doxes in physics and math that can be dis­solved. Read­ing this book is 100% bet­ter than con­tin­u­ing to go through your life with a hazy un­der­stand­ing of how im­por­tant things like free will, choice, and mean­ing ac­tu­ally work.

I recom­mend read­ing this right around the time you finish up your quan­tum com­put­ing course.
MIRI Re­search Papers
MIRI has already pub­lished 30+ re­search pa­pers that can help ori­ent fu­ture Friendli­ness re­searchers. The work is pretty fas­ci­nat­ing and read­ily ac­cessible for peo­ple in­ter­ested in the sub­ject. For ex­am­ple: How do differ­ent pro­pos­als for value ag­gre­ga­tion and ex­trap­o­la­tion work out? What are the likely out­comes of differ­ent in­tel­li­gence ex­plo­sion sce­nar­ios? Which eth­i­cal the­o­ries are fit for use by an FAI? What im­prove­ments can be made to mod­ern de­ci­sion the­o­ries to stop them from di­verg­ing from win­ning strate­gies? When will AI ar­rive? Do AIs de­serve moral con­sid­er­a­tion? Even though most of your work will be more tech­ni­cal than this, you can still gain a lot of shared back­ground knowl­edge and more clearly see where the broad prob­lem space is lo­cated.

I’d recom­mend read­ing these any­time af­ter you finish read­ing The Se­quences and Global Catas­trophic Risks.
Univer­sal Ar­tifi­cial Intelligence
A use­ful book on “op­ti­mal” AI that gives a rea­son­able for­mal­ism for study­ing how the most pow­er­ful classes of AIs would be­have un­der con­ser­va­tive safety de­sign sce­nar­ios (i.e., lots and lots of rea­son­ing abil­ity).

Wait un­til you finish most of the course­work above be­fore try­ing to tackle this one.

Do also look into: For­mal Episte­mol­ogy, Game The­ory, De­ci­sion The­ory, and Deep Learn­ing.