Recommended Reading for Friendly AI Research
This post enumerates texts that I consider (potentially) useful training for making progress on Friendly AI/decision theory/metaethics.
Rationality and Friendly AI
Eliezer Yudkowsky’s sequences and this blog can provide solid introduction to the problem statement of Friendly AI, giving concepts useful for understanding motivation for the problem, and disarming endless failure modes that people often fall into when trying to consider the problem.
For a shorter introduction, see
Eliezer S. Yudkowsky (2008). “Artificial Intelligence as a Positive and Negative Factor in Global Risk”. Global Catastrophic Risks. Oxford University Press.
Decision theory
The following book introduces an approach to decision theory that seems to be closer to what’s needed for FAI than the traditional treatments in philosophy or game theory:
G. L. Drescher (2006). Good and Real: Demystifying Paradoxes from Physics to Ethics (Bradford Books). The MIT Press, 1 edn.
Another (more technical) treatment of decision theory from the same cluster of ideas:
E. Yudkowsky. Timeless Decision Theory (draft, Sep 2010)
Following posts on Less Wrong present ideas relevant to this development of decision theory:
- A Priori
- Newcomb’s Problem and Regret of Rationality
- The True Prisoner’s Dilemma
- Counterfactual Mugging
- Timeless Decision Theory: Problems I Can’t Solve
- Towards a New Decision Theory
- Ingredients of Timeless Decision Theory
- Decision theory: Why Pearl helps reduce “could” and “would”, but still leaves us with at least three alternatives
- The Absent-Minded Driver
- AI cooperation in practice
- What a reduction of “could” could look like
- Controlling Constant Programs
- Notion of Preference in Ambient Control
Mathematics
The most relevant tool for thinking about FAI seems to be mathematics, where it teaches to work with precise ideas (in particular, mathematical logic). Starting from a rusty technical background, the following reading list is one way to start:
[Edit Nov 2011: I no longer endorse scope/emphasis, gaps between entries, and some specific entries on this list.]
F. W. Lawvere & S. H. Schanuel (1991). Conceptual mathematics: a first introduction to categories. Buffalo Workshop Press, Buffalo, NY, USA.
B. Mendelson (1962). Introduction to Topology. College Mathematics. Allyn & Bacon Inc., Boston.
P. R. Halmos (1960). Naive Set Theory. Springer, first edn.
H. B. Enderton (2001). A Mathematical Introduction to Logic. Academic Press, second edn.
S. Mac Lane & G. Birkhoff (1999). Algebra. American Mathematical Society, 3 edn.
F. W. Lawvere & R. Rosebrugh (2003). Sets for Mathematics. Cambridge University Press.
J. R. Munkres (2000). Topology. Prentice Hall, second edn.
- S. Awodey (2006). Category Theory. Oxford Logic Guides. Oxford University Press, USA.
K. Kunen (1999). Set Theory: An Introduction To Independence Proofs, vol. 102 of Studies in Logic and the Foundations of Mathematics. Elsevier Science, Amsterdam.
- P. G. Hinman (2005). Fundamentals of Mathematical Logic. A K Peters Ltd.
- References & Resources for LessWrong by (10 Oct 2010 14:54 UTC; 162 points)
- Lesswrong Community’s How-Tos and Recommendations by (7 May 2012 13:41 UTC; 38 points)
- 's comment on Deontological Decision Theory and The Solution to Morality by (10 Jan 2011 16:50 UTC; 20 points)
- 's comment on Secrets of the eliminati by (30 Jul 2011 16:28 UTC; 8 points)
- How to better understand and participate on LW by (8 Oct 2010 16:11 UTC; 8 points)
- 's comment on Course recommendations for Friendliness researchers by (23 Jan 2013 0:10 UTC; 8 points)
- 's comment on Less Wrong Rationality and Mainstream Philosophy by (21 Mar 2011 20:14 UTC; 5 points)
- 's comment on Which are the useful areas of AI study? by (16 Jan 2011 1:06 UTC; 4 points)
- 's comment on Friendly AI Research and Taskification by (14 Dec 2010 11:46 UTC; 3 points)
- 's comment on LW is to rationality as AIXI is to intelligence by (7 Mar 2011 1:20 UTC; 3 points)
- 's comment on Reaching young math/compsci talent by (17 Jun 2012 9:52 UTC; 2 points)
- 's comment on David Chalmers’ “The Singularity: A Philosophical Analysis” by (29 Jan 2011 22:19 UTC; 1 point)
- 's comment on What a reduction of “could” could look like by (18 Feb 2012 19:21 UTC; 0 points)
- 's comment on Hard problem? Hack away at the edges. by (28 Sep 2011 10:02 UTC; 0 points)
- 's comment on Best career models for doing research? by (8 Dec 2010 13:00 UTC; -1 points)
Thanks for posting this. Some things to add to my reading list.
If you consider this “(potentially) useful training for making progress on Friendly AI”, then do you expect that a person who has worked through this material will have a good sense of whether they are qualified to actually try to make progress on FAI (or will be evaluable for that by someone with more experience working on FAI)? I want to do as much as I can to contribute to FAI, whether directly (by working on the actual problems) or indirectly (e.g. getting rich and donating a lot to SIAI), whichever I end up being most efficient at. Right now I’m not efficient at much of anything, because of some severe issues with mental energy that I’m only now starting to possibly resolve after several years, but once I am more competent at life in general, I want to at least investigate the possibility that I could be directly useful to FAI research. (I’m not a savant or a mutant supergenius, but I am at least a normal genius.) If, at that point, I can get through all of this math successfully, will that be an indication that I should look further?
My best guess at productive subgoal for FAI is development of decision theory along the lines given in the last post, in order to better understand decision-making and the impossible problem in particular (how to define preference given an arbitrary agent’s program; what is a notion of preference that is general enough for human preference to be an instance).
About a year ago I was still at the “rusty technical background” stage, and my attempts to think about decision theory were not quite adequate. Studying mathematics helped significantly by allowing to think more clearly and about more complicated constructions. More recently, study of mathematical logic allowed me to see the beautiful formalizations of decision theory I’m currently working on.
I can’t tell you that studying this truckload of textbooks will get any results, but reading textbooks is something I know how to do, unlike how to make progress on FAI, so unless I find something better, it’s what I’ll continue doing.
Ambient decision theory, as it currently stands, requires some grasp of logic to think about, but the level of Enderton’s book might be adequate. I’m going deeper in the hope of developing more mathematical muscle to allow jumping over wider inferential gaps, even if I don’t know in what way. Relying on creative surprises.
Interesting… I had missed Eliezer’s draft Timeless Decision Theory before. Has it been discussed already?
I first saw its mention in this Open Thread, followed by a small discussion.
It might be helpful for those who have learned a significant amount of mathematics to share their insights about how to learn math. Though individuals may have different learning patterns, I expect that some common principles and practices would arise from a focused discussion. This could increase the rate of productive learning.
ETA: I’ve made a discussion post where we can, um, discuss.
Are there any parts of theoretical computer science that you expect will be useful?
If you think the subject matter of mathematical logic will be useful then it might be worthwhile to take a look at the Curry-Howard correspondence. From the link:
Another book to consider: The Princeton Companion to Mathematics
It addresses fundamental mathematical ideas, theories, and practices from an insightful conceptual perspective. Using it one could build a large connected web of mathematical concepts, which would likely be useful for identifying the mathematical structure in new problems.
Also, IMO the writing quality is outstanding. The editor went out of his way to find mathematicians that were also excellent writers (like Terence Tao).
It could be useful to (already) know many things, but another question is how to efficiently get to learn them, starting from what background. Once you are at graduate level, a lot more becomes accessible, so the first step is to get there.
My sequence meant to suggest a way of reaching that level by self-study, and getting a good grasp of basic tools of logic in the process. It’s probably not the best way, but if you suggest improvements, they ought to be improvements in achieving this particular goal, not just things associated with elements of the original plan, such as “more math books”. Other goal could be worthwhile too, but it would be better to state the different intention before proceeding.
Ah, I see. You could make that more clear in the post fairly easily.
It might be helpful to suggest two paths of self-study, targeting those below and at graduate-level respectively. I’m not sure if that would be best done in a separate post or not.
A suggested order might be useful. I’d at least recommend reading about algebra and topology before category theory, so that one builds up fundamental examples of category-theoretic objects.
The books are suggested in order as given.
I essentially followed this rule. “Conceptual mathematics” is elementary, “Sets for mathematics” deals mainly with set theory from category-theoretic perspective. The more general treatment of category theory is given in Awodey’s book, which comes after Munkres that presents general and algebraic topology. Mac Lane’s algebra comes before “Sets for mathematics” as well.
I am sure these are interesting references for studying pure mathematics but do they contribute significantly to solving AI?
In particular, it is interesting that none of your references mention any existing research on AI. Are there any practical artificial intelligence problems that these mathematical ideas have directly contributed towards solving?
E.g. Vision, control, natural language processing, automated theorem proving?
While there is a lot of focus on specific, mathematically defined problems on LessWrong (usually based on some form of gambling), there seems to be very little discussion of the actual technical problems of GAI or a practical assessment of progress towards solving them. If this site is really devoted to rationality should we not at least define our problem and measure progress towards its solution. Otherwise we risk being merely a mathematical social club, or worse, a probability based religion?
The main mystery in FAI, as I currently see it, is how to define its goal. The question of efficient implementation comes after that and depending on that. There is no point in learning how to efficiently solve the problem you don’t want to be solved. Hence the study of decision theory, which in turn benefits from understanding math.
See the “rationality and FAI” section, Eliezer’s paper for a quick introduction, also stuff from sequences, for example complexity of value.
Ok, I certainly agree that defining the goal is important. Although I think there is a definite need for a balance between investigation of the problem and attempts at its solution (as each feed into one another). Much as how academia currently functions. For example, any AI will need a model of human and social behaviour in order to make predictions. Solving how an AI might learn this would represent a huge step towards solving FAI and a huge step in understanding the problem of being friendly. I.e. whatever the solution is will involve some configuration of society that maintains and maximises some set of measurable properties from it.
If the system can predict how a person will feel in a given state it can solve for which utopia we will be most enthusiastic about. Eliezer’s posts seem to be exploring this problem manually, without really taking a stab at a solution, or proposing a route to reaching one. This can be very entertaining but I’m not sure it’s progress.
Unfortunately, if you think about it, “predicting how a person feels” isn’t really helpful to anything, and doesn’t contribute to the project of FAI at all (see Are wireheads happy? and The Hidden Complexity of Wishes, for example).
The same happens with other obvious ideas that you think up in the first 5 minutes of considering the problem, and which appear to argue that “research into nuts and bolts of AGI” is relevant for FAI. But on further reflection, it always turns out that these arguments don’t hold any water.
The problem comes down the the question of understanding of what it is exactly you want FAI to do, not of how you’d manage to write an actual program that does that with reasonable efficiency. The horrible truth is that we don’t have the slightest technical understanding of what it is we want.
Here is a more complex variant that I can’t see how to dismiss easily.
If you can build a “predict how humans feel in situation x” function, you can do some interesting things. Lets call this function feel(x). Now as well as first order happiness, you can also predict how they will feel when told about situation X, so feel(“told about X”).
You might be able to recover something like preference if you can calculate feel(“the situation where X is suggested and, told about feel(X) and told about all other possible situations”) , for all possible situations, as long as you can rank the output of feel(X) in some way.
Well as long as the human simulator predictor can cope with holding in all possible situations, and not return “worn out” for all situations.
Anyway it is an interesting riff off the idea. Anyone see any holes that I am missing?
Try to figure out what maximizes this estimate method. It won’t be anything you’d want implemented, it will be a wireheading stimulus. Plus, FAI needs to valuate (and work to implement) whole possible worlds, not verbal descriptions. And questions about possible worlds involve quantifies of data that a mere human can’t handle.
I’m not sure that there is a verbal description of a possible world that is also a wirehead stimulus for me. There might be, which might be enough to discount this method.
True.
I’m not sure I understand the distinction between an answer that we would want and a wireheading solution. Are not all solutions wireheading with an elaborate process to satisfy our status concerns. I.e. is there a real difference between a world that satisfies what we want and directly altering what we want? If the wire in question happens to be an elaborate social order rather than a direct connection why is that different? What possible goal could we want pursued other than the one which we want?
From an evolutionary point of view those things that manage to procreate will out compete those things that change themselves to not care about that and just wirehead.
So in non-singleton situations, alien encounters and any form of resource competition it matters whether you wirehead or not. Pleasure, in an evolved creature, can be seen as the giving (very poor) information on the map to the territory of future influence for the patterns that make up you.
So, assuming survival is important, a solution that maximises survival plus wireheading would seem to solve that problem. Of course it may well just delay the inevitable heat death ending but if we choose to make that important, then sure, we can optimise for survival as well. I’m not sure that gets around the issue that any solution we produce (with or without optimisation for survival) is merely an elaborate way of satisfying our desires (in this case including the desire to continue to exist) and thus all FAI solutions are a form of wireheading.
When I say feel, I include:
I feel that is correct. I feel that is proved etc.
Regardless of the answer, it will ultimately involve our minds expressing a preference. We cannot escape our psychology. If our minds are deterministic computational machines within a universe without any objective value, all our goals are merely elaborate ways to make us feel content with our choices and a possibly inconsistent set of mental motivations. Attempting to model our psychology seems like the most efficient way to solve this problem. Is the idea that there is some other kind of answer? How would could it be shown to be legitimate?
I suspect that the desire for another answer is preventing practical progress in creating any meaningful solution. There are many problems and goals that would be relatively uncontroversial for an AI system to attempt to address. The outcome of the work need only be better than what we currently have to be useful we don’t have to solve all problems before addressing some of them and indeed without attempting to address some of them I doubt we will make significant progress on the rest.
Which problem? You need to define which action should AI choose, in whatever problem it’s solving, including the problems that are not humanly comprehensible. This is naturally done in terms of actual humans with all their psychology (as the only available source of sufficiently detailed data about what we want), but it’s not at all clear in what way you’d want to use (interpret) that human data.
“Attempting to model psychology” doesn’t answer any questions. Assume you have a proof-theoretic oracle and a million functioning uploads living in a virtual world however structured, so that you can run any number of experiments involving them, restart these experiments, infer the properties of whole infinite collections of such experiments and so on. You still won’t know how to even approach creating a FAI.
If there is an answer to the problem of creating an FAI, it will result from a number of discussions and ideas that lead a set of people to agreeing that a particular course of action is a good one. By modelling psychology it will be possible to determine all the ways this can be done. The question then is why choose one over any of the others? As soon as one is chosen it will work and everyone will go along with it. How could we rate each one? (they would all be convincing by definition). Is it meaningful to compare them? Is the idea that there is some transcendent answer that is correct or important that doesn’t boil down to what is convincing to people?
Understanding the actual abstract reasons for agents’ decisions (such as decisions about agreeing with a given argument) seems to me a promising idea, I’m trying to make progress on that (agents’ decisions don’t need to be correct or well-defined on most inputs for the reasons behind their more well-defined behaviors to lead the way to figuring out what to do in other situations or what should be done where the agents err). Note that if you postulate an algorithm that makes use of humans as its elements, you’d still have the problems of failure modes, regret for bad design decisions and of the capability to answer humanly incomprehensible questions, and these problems need to be already solved before you start the thing up.
Interesting, if I understand correctly the idea is to find a theoretically correct basis for deciding on a course of action given existing knowledge and then to make this calculation efficient and then direct towards a formally defined objective.
As distinct from a system which potentially sub optimally, attempts solutions and tries to learn improved strategies. i.e. one in which the theoretical basis for decision making is ultimately discovered by the agent over time (e.g. as we have done with the development of probability theory). I think the perspective I’m advocating is to produce a system that is more like an advanced altruistic human (with a lot of evolutionary motivations removed) than a provably correct machine. Ideally such a system could itself propose solutions to the FAI problem that would be convincing, as a result of an increasingly sophisticated understanding of human reasoning and motivations.
I realise there is a fear that such a system could develop convincing yet manipulative solutions. However the output need only be more trustworthy than a human’s response to be legitimate (for example based on an analysis of its reasoning algorithm it appears to lack a Machiavellian capability, unlike humans).
Or put another way, can a robot Vladimir (Eliezer etc.) be made that solves the problem faster than their human counterparts do. And is there any reason to think this process is less safe (particularly when AI developments will continue regardless)?
Yes, but there is only one top-level objective, to do the right thing, so one doesn’t need to define an objective separately from the goal system itself (and improving state of knowledge is just another thing one can do to accomplish the goal, so again not a separate issue).
FAI really stands for a method of efficient production of goodness, as we would want it produced, and there are many landmines on this path, in particular humanity in its current form doesn’t seem to be able to retain its optimization goal in the long run, and the same applies to most obvious hacks that don’t have explicit notions of preference, such as upload societies. It’s not just a question of speed, but also of ability to retain the original goal after quadrillions of incompletely understood self-modifications.
Ok, so how about this work around.
The current approach is to have a number of human intelligences continue to explore this problem until they enter a mental state C (for convinced they have the answer to FAI). The next stage is to implement it.
We have no other route to knowledge other than to use our internal sense of being convinced. I.e. no oracle to tell us if we are right or not.
So what if we formally define what this mental state C consists of and then construct a GAI which provably pursues only the objective of creating this state. The advantage being that we now have a means of judging our progress because we have a formally defined measurable criteria for success. (In fact this process is a valuable goal regardless of the use of AI but it now makes it possible to use AI techniques to solve it).
Related by Cousin It: http://lesswrong.com/r/discussion/lw/2sw/math_prerequisites_for_understanding_lw_stuff/
It would be helpful if the books we ordered by dependency (topological sort) & order of difficulty .
Don’t forget nickbostrom.com and Moral Machines: Teaching Robots Right from Wrong.