This is placed in the context of Rohin’s thesis that solving AI alignment is best achieved by designing AI which is not goal-directed.
[...]
I remain skeptical about Rohin’s thesis that we should dispense with goal-directedness altogether
Hmm, perhaps I believed this when I wrote the sequence (I don’t think so, but maybe?), but I certainly don’t believe it now. I believe something more like:
Humans have goals and want AI systems to help them achieve them; this implies that the human-AI system as a whole should be goal-directed.
One particular way to do this is to create a goal-directed AI system, and plug in a goal that (we think) we want. Such AI systems are well-modeled as EU maximizers with “simple” utility functions.
But there could plausibly be AI systems that are not themselves goal-directed, but nonetheless the resulting human-AI system is sufficiently goal-directed. For example, a “genie” that properly interprets your instructions based on what you mean and not what you say seems not particularly goal-directed, but when combined with a human giving instructions becomes goal-directed.
One counterargument is that in order to be competitive, you must take the human out of the loop. I don’t find this compelling, for a few reasons. First, you can interpolate between lots of human feedback (the human says “do X for a minute” every minute to the “genie”) and not much human feedback (the human says “pursue my CEV forever”) depending on how competitive you need to be. This allows you to tradeoff between competitiveness and how much of the goal-directedness remains in the human. Second, you can help the human to provide more efficient and effective feedback (see e.g. recursive reward modeling). Finally, laws and regulations can be effective at reducing competition.
Nonetheless, it’s not obvious how to create such non-goal-directed AI, and the AI community seems very focused on building goal-directed AI, and so there’s a good chance we will build goal-directed AI and will need to focus on alignment of goal-directed AI systems.
As a result, we should be thinking about non-goal-directed AI approaches to alignment, while also working on alignment of goal-directed systems.
I think when I wrote the sequence, I thought the “just do deep RL” approach to AGI wouldn’t work, and now I think it has more of a chance, and this has updated me towards powerful AI systems being goal-directed. (However, I do not think it is clear that “just do deep RL” approaches lead to goal-directed systems.)
I basically disagree with the recommendation almost always, including for AI alignment. I do think that
I often see the sentiment, “I’m going to learn linear algebra, probability theory, computational complexity, machine learning and deep RL, and then I’ll have the prerequisites to do AI safety”. (Possible reasons for this: the 80K AI safety syllabus, CHAI’s bibliography, a general sense that you have to be an expert before you can do research.) This sentiment seems wrong to me; you definitely can and should think about important questions before learning everything that could potentially be considered “background”.
The advice
sounds to me like “when you feel like existing research would be useful, then go ahead and look at it, but don’t feel like it’s necessary”, whereas I would say “as soon as you have questions, which should be almost immediately, one of the first things you should do is find the existing research and read it”. The justification for this is the standard one—people have already done a bunch of work that you can take advantage of.
The main disadvantage of this approach is that you lose the opportunity to figure things out from first principles. When you figure things out from first principles, you often find many branches that don’t work out, which helps build intuitions about why things are the way they are, which you don’t get nearly as well by reading about research, and you can’t go back and figure things out from first principles because you already know the answer. But this first-principles-reasoning is extremely expensive (in time), and is almost never worthwhile.
Another potential disadvantage is that you might be incorrectly convinced that a technique is good, because you don’t spot the flaws in it when reading existing research, even though you could have figured it out from first principles. My preferred solution is to become good at noticing flaws (e.g. by learning how to identify and question all of the assumptions in an argument), rather than to ignore research entirely.
Side note: In the case of philosophy, if you’re trying to get a paper, then I’m told you often want to make some novel argument (since that’s what gets published), which makes existing research less useful (or only useful to figure out what not to think about). If you want to figure out the truth, I expect you would do well to read existing research.
TL;DR: Looking at existing research is great because you don’t have to reinvent the wheel, but make sure you need the wheel in the first place before you read about it (i.e. make sure you have a question you are reading existing research to answer).
ETA: If your goal is “maximize understanding of X”, then you should never look at existing research about X, and figure everything out from first principles. I’m assuming that you have some reason for caring about X that means you are willing to trade off some understanding for getting it done way faster.