Course recommendations for Friendliness researchers

When I first learned about Friendly AI, I assumed it was mostly a programming problem. As it turns out, it’s actually mostly a math problem. That’s because most of the theory behind self-reference, decision theory, and general AI techniques haven’t been formalized and solved yet. Thus, when people ask me what they should study in order to work on Friendliness theory, I say “Go study math and theoretical computer science.”

But that’s not specific enough. Should aspiring Friendliness researchers study continuous or discrete math? Imperative or functional programming? Topology? Linear algebra? Ring theory?

I do, in fact, have specific recommendations for which subjects Friendliness researchers should study. And so I worked with a few of my best interns at MIRI to provide recommendations below:

  • University courses. We carefully hand-picked courses on these subjects from four leading universities — but we aren’t omniscient! If you’re at one of these schools and can give us feedback on the exact courses we’ve recommended, please do so.

  • Online courses. We also linked to online courses, for the majority of you who aren’t able to attend one of the four universities whose course catalogs we dug into. Feedback on these online courses is also welcome; we’ve only taken a few of them.

  • Textbooks. We have read nearly all the textbooks recommended below, along with many of their competitors. If you’re a strongly motivated autodidact, you could learn these subjects by diving into the books on your own and doing the exercises.

Have you already taken most of the subjects below? If so, and you’re interested in Friendliness research, then you should definitely contact me or our project manager Malo Bourgon (malo@intelligence.org). You might not feel all that special when you’re in a top-notch math program surrounded by people who are as smart or smarter than you are, but here’s the deal: we rarely get contacted by aspiring Friendliness researchers who are familiar with most of the material below. If you are, then you are special and we want to talk to you.

Not everyone cares about Friendly AI, and not everyone who cares about Friendly AI should be a researcher. But if you do care and you might want to help with Friendliness research one day, we recommend you consume the subjects below. Please contact me or Malo if you need further guidance. Or when you’re ready to come work for us.

COGSCI C127
PHIL 190
6.804J
85-213
Free Online
Cognitive Science
If you’re endeavoring to build a mind, why not start by studying your own? It turns out we know quite a bit: human minds are massively parallel, highly redundant, and although parts of the cortex and neocortex seem remarkably uniform, there are definitely dozens of special purpose modules in there too. Know the basic details of how the only existing general purpose intelligence currently functions.
ECON 119
IPS 207A
15.847
80-302
Free Online
Heuristics and Biases
While cognitive science will tell you all the wonderful things we know about the immense, parallel nature of the brain, there’s also the other side of the coin. Evolution designed our brains to be optimized at doing rapid thought operations that work in 100 steps or less. Your brain is going to make stuff up to cover up that its mostly cutting corners. These errors don’t feel like errors from the inside, so you’ll have to learn how to patch the ones you can and then move on.

PS—We should probably design our AIs better than this.
COMPSCI 61A
MATH 198
6.005
15-150
Free Online
Functional Programing
There are two major branches of programming: Functional and Imperative. Unfortunately, most programmers only learn imperative programming languages (like C++ or python). I say unfortunately, because these languages achieve all their power through what programmers call “side effects”. The major downside for us is that this means they can’t be efficiently machine checked for safety or correctness. The first self-modifying AIs will hopefully be written in functional programming languages, so learn something useful like Haskell or Scheme.
MATH 55
CME 305
6.042J/​18.062J
21-228
Free Online
Discrete Math
Much like programming, there are two major branches of mathematics as well: Discrete and continuous. It turns out a lot of physics and all of modern computation is actually discrete. And although continuous approximations have occasionally yielded useful results, sometimes you just need to calculate it the discrete way. Unfortunately, most engineers squander the majority of their academic careers studying higher and higher forms of calculus and other continuous mathematics. If you care about AI, study discrete math so you can understand computation and not just electricity.

Also, you should pick up enough graph theory in this course to handle the basic mechanics of decision theory—which you’re gonna want to learn later.
MATH 110
MATH 113
18.06
21-341
Free Online
Linear Algebra
Linear algebra is the foundation of quantum physics and a huge amount of probability theory. It even shows up in analyses of things like neural networks. You can’t possibly get by in machine learning (later) without speaking linear algebra. So learn it early in your scholastic career.
MATH 135
MATH 161
24.243
21-602
Free Online
Set Theory
Like learning how to read in mathematics. But instead of building up letters into words, you’ll be building up axioms into theorems. This will introduce you to the program of using axioms to capture intuition, finding problems with the axioms, and fixing them.
MATH 125A
CS 103
24.241
21-600
Free Online
Mathematical Logic
COMPSCI 170
CS 161
6.046J
15-451
Free Online
Efficient Algorithms and Intractable Problems
Like building sentences into paragraphs. Algorithms are the recipes of thought. One of the more amazing things about algorithm design is that it’s often possible to tell how long a process will take to solve a problem before you actually run the process to check it. Learning how to design efficient algorithms like this will be a foundational skill for anyone programming an entire AI, since AIs will be built entirely out of collections of algorithms.
MATH 128A
CME206
18.330
21-660
Free Online
Numerical Analysis
There are ways to systematically design algorithms that only get things slightly wrong when the input data has tiny errors. And then there’s programs written by amateur programmers who don’t take this class. Most programmers will skip this course because it’s not required. But for us, getting the right answer is very much required.
COMPSCI 172
CS 154
6.840J
15-453
Free Online
Computability and Complexity
This is where you get to study computing at it’s most theoretical. Learn about the Church-Turing thesis, the universal nature and applicability of computation, and how just like AIs, everything else is algorithms… all the way down.
COMPSCI 191
CS 259Q
6.845
33-658
Free Online
Quantum Computing
It turns out that our universe doesn’t run on Turing Machines, but on quantum physics. And something called BQP is the class of algorithms that are actually efficiently computable in our universe. Studying the efficiency of algorithms relative to classical computers is useful if you’re programming something that only needs to work today. But if you need to know what is efficiently computable in our universe (at the limit) from a theoretical perspective, quantum computing is the only way to understand that.
COMPSCI 273
CS149
18.337J
15-418
Free Online
Parallel Computing
There’s a good chance that the first true AIs will have at least some algorithms that are inefficient. So they’ll need as much processing power as we can throw at them. And there’s every reason to believe that they’ll be run on parallel architectures. There are a ton of issues that come up when you switch from assuming sequential instruction ordering to parallel processing. There’s threading, deadlocks, message passing, etc. The good part about this course is that most of the problems are pinned down and solved: You’re just learning the practice of something that you’ll need to use as a tool, but won’t need to extend much (if any).
EE 219C
MATH 293A
6.820
15-414
Free Online
Automated Program Verification
Remember how I told you to learn functional programming way back at the beginning? Now that you wrote your code in functional style, you’ll be able to do automated and interactive theorem proving on it to help verify that your code matches your specs. Errors don’t make programs better and all large programs that aren’t formally verified are reliably *full* of errors. Experts who have thought about the problem for more than 5 minutes agree that incorrectly designed AI could cause disasters, so world-class caution is advisable.
COMPSCI 174
CS 109
6.042J
21-301
Free Online
Combinatorics and Discrete Probability
Life is uncertain and AIs will handle that uncertainty using probabilities. Also, probability is the foundation of the modern concept of rationality and the modern field of machine learning. Probability theory has the same foundational status in AI that logic has in mathematics. Everything else is built on top of probability.
STAT 210A
STATS 270
6.437/​438
36-266
Free Online
Bayesian Modeling and Inference
Now that you’ve learned how to calculate probabilities, how do you combine and compare all the probabilistic data you have? Like many choices before, there is a dominant paradigm (frequentism) and a minority paradigm (Bayesianism). If you learn the wrong method here, you’re deviating from a knowably correct framework for integrating degrees of belief about new information and embracing a cadre of special purpose, ad-hoc statistical solutions that often break silently and without warning. Also, quite embarrassingly, frequentism’s ability to get things right is bounded by how well it later turns out to have agreed with Bayesian methods anyway. Why not just do the correct thing from the beginning and not have your lunch eaten by Bayesians every time you and them disagree?
MATH 218A/​B
MATH 230A/​B/​C
6.436J
36-225/​21-325
Free Online
Probability Theory
No more applied probability: Here be theory! Deep theories of probabilities are something you’re going to have to extend to help build up the field of AI one day. So you actually have to know why all the things you’re doing are working inside out.
COMPSCI 189
CS 229
6.867
10-601
Free Online
Machine Learning
Now that you chose the right branch of math, the right kind of statistics, and the right programming paradigm, you’re prepared to study machine learning (aka statistical learning theory). There are lots of algorithms that leverage probabilistic inference. Here you’ll start learning techniques like clustering, mixture models, and other things that cache out as precise, technical definitions of concepts that normally have rather confused or confusing English definitions.
COMPSCI 188
CS 221
6.034
15-381
Free Online
Artificial Intelligence
We made it! We’re finally doing some AI work! Doing logical inference, heuristic development, and other techniques will leverage all the stuff you just learned in machine learning. While modern, mainstream AI has many useful techniques to offer you, the authors will tell you outright that, “the princess is in another castle”. Or rather, there isn’t a princess of general AI algorithms anywhere—not yet. We’re gonna have to go back to mathematics and build our own methods ourselves.
MATH 136
PHIL 152
18.511
80-311
Free Online
Incompleteness and Undecidability
Probably the most celebrated results is mathematics are the negative results by Kurt Goedel: No finite set of axioms can allow all arithmetic statements to be decided as either true or false… and no set of self-referential axioms can even “believe” in its own consistency. Well, that’s a darn shame, because recursively self-improving AI is going to need to side-step these theorems. Eventually, someone will unlock the key to over-coming this difficulty with self-reference, and if you want to help us do it, this course is part of the training ground.
MATH 225A/​B
PHIL 151
18.515
21-600
Free Online
Metamathematics
Working within a framework of mathematics is great. Working above mathematics—on mathematics—with mathematics, is what this course is about. This would seem to be the most obvious first step to overcoming incompleteness somehow. Problem is, it’s definitely not the whole answer. But it would be surprising if there were no clues here at all.
MATH 229
MATH 290B
24.245
21-603
Free Online
Model Theory
One day, when someone does side-step self-reference problems enough to program a recursively self-improving AI, the guy sitting next to her who glances at the solution will go “Gosh, that’s a nice bit of Model Theory you got there!”

Think of Model Theory as a formal way to understand what “true” means.
MATH 245A
MATH 198
18.996
80-413
Free Online
Category Theory
Category theory is the precise way that you check if structures in one branch of math represent the same structures somewhere else. It’s a remarkable field of meta-mathematics that nearly no one knows… and it could hold the keys to importing useful tools to help solve dilemmas in self-reference, truth, and consistency.
Outside recommendations
Harry Potter and the Methods of Rationality
Global Catastrophic Risks
A good primer on xrisks and why they might matter. SPOILER ALERT: They matter.

You can probably skim read this early on in your studies. Right after HP:MoR.
The Sequences
Rationality: the indispensable art of non-self-destruction! There are manifold ways you can fail at life… especially since your brain is made out of broken, undocumented spaghetti code. You should learn more about this ASAP. That goes double if you want to build AIs.

I highly recommend you read this before you get too deep into your academic career. For instance, I know people who went to college for 5 years, while somehow managing to learn nothing. That’s because instead of learning, they merely recited the teacher’s password every semester until they could dump whatever they “learned” out of their heads as soon as they walked out of the final. Don’t let this happen to you! This, and a hundred other useful lessons like it about how to avoid predictable, universal errors in human reasoning and behavior await you in The Sequences!
Good and Real
A surprisingly thoughtful book on decision theory and other paradoxes in physics and math that can be dissolved. Reading this book is 100% better than continuing to go through your life with a hazy understanding of how important things like free will, choice, and meaning actually work.

I recommend reading this right around the time you finish up your quantum computing course.
MIRI Research Papers
MIRI has already published 30+ research papers that can help orient future Friendliness researchers. The work is pretty fascinating and readily accessible for people interested in the subject. For example: How do different proposals for value aggregation and extrapolation work out? What are the likely outcomes of different intelligence explosion scenarios? Which ethical theories are fit for use by an FAI? What improvements can be made to modern decision theories to stop them from diverging from winning strategies? When will AI arrive? Do AIs deserve moral consideration? Even though most of your work will be more technical than this, you can still gain a lot of shared background knowledge and more clearly see where the broad problem space is located.

I’d recommend reading these anytime after you finish reading The Sequences and Global Catastrophic Risks.
Universal Artificial Intelligence
A useful book on “optimal” AI that gives a reasonable formalism for studying how the most powerful classes of AIs would behave under conservative safety design scenarios (i.e., lots and lots of reasoning ability).

Wait until you finish most of the coursework above before trying to tackle this one.

Do also look into: Formal Epistemology, Game Theory, Decision Theory, and Deep Learning.