Former AI safety research engineer, now AI governance researcher at OpenAI. Blog: thinkingcomplete.blogspot.com
Richard_Ngo
Karma: 8,040
Thanks for the comments Vika! A few responses:
It might be good to clarify that this is an example architecture and the claims apply more broadly.
Makes sense, will do.
Phase 1 and 2 seem to map to outer and inner alignment respectively.
That doesn’t quite seem right to me. In particular:
Phase 3 seems like the most direct example of inner misalignment; I basically think of “goal misgeneralization” as a more academically respectable way of talking about inner misalignment.
Phase 1 introduces the reward misspecification problem (which I treat as synonymous with “outer alignment”) but also notes that policies might become misaligned by the end of phase 1 because they learn goals which are “robustly correlated with reward because they’re useful in a wide range of environments”, which is a type of inner misalignment.
Phase 2 discusses both policies which pursue reward as an instrumental goal (which seems more like inner misalignment) and also policies which pursue reward as a terminal goal. The latter doesn’t quite feel like a central example of outer misalignment, but it also doesn’t quite seem like a central example of reward tampering (because “deceiving humans” doesn’t seem like an example of “tampering” per se). Plausibly we want a new term for this—the best I can come up with after a few minutes’ thinking is “reward fixation”, but I’d welcome alternatives.
Supposing there is no misspecification in phase 1, do the problems in phase 2 still occur? How likely is deceptive alignment seems to argue that they may not occur, since a model that has perfect proxies when it becomes situationally aware would not then become deceptively aligned.
It seems very unlikely for an AI to have perfect proxies when it becomes situationally aware, because the world is so big and there’s so much it won’t know. In general I feel pretty confused about Evan talking about perfect performance, because it seems like he’s taking a concept that makes sense in very small-scale supervised training regimes, and extending it to AGIs that are trained on huge amounts of constantly-updating (possibly on-policy) data about a world that’s way too complex to predict precisely.
I’m confused why mechanistic interpretability is listed under phase 3 in the research directions—surely it would make the most difference for detecting the emergence of situational awareness and deceptive alignment in phase 2, while in phase 3 the deceptively aligned model will get around the interpretability techniques.
Mechanistic interpretability seems helpful in phase 2, but there are other techniques that could help in phase 2, in particular scalable oversight techniques. Whereas interpretability seems like the only thing that’s really helpful in phase 3 - if it gets good enough then we’ll be able to spot agents trying to “get around” our techniques, and/or intervene to make their concepts generalize in more desirable ways.
even if have empirical evidence for the problem being “level n hard” (people have tried up to level n), you;d still do not have empirical evidence for the problem being “level n+1 hard”
This is implicitly assuming that our expectation of how long a problem should take to solve is memoryless. But a breakthrough is much more likely on the 1st day of working on a problem than on the 1000th day. More generally, if problems vary greatly in difficulty, then our failure to solve a given problem provides evidence that it’s one of the harder problems. So a more reasonable prior in this case is something like logarithmic—e.g. it’s equally likely that a problem takes 1-10 days, or 10-100 days, or 100-1000 days, etc, to solve.
A similar model can give rise to the Lindy effect, where the expected lifetime is proportional to the lifetime so far. (In this case it’d be the expected time to solving the problem which would be proportional to the time which the problem has been open.)
Thanks! Fixed, although I feel confused about whether the second is actually grammatically incorrect or not (you could interpret it as “papers such that it’s good to replicate those papers”, but idk what the rulebook would say).
How? E.g. Jacob left a comment here about his motivations, does that count as a falsification? Or, if you’d say that this is an example of rationalization, then what would the comment need to look like in order to falsify your claim? Does Paul’s comment here mentioning the discussions that took place before launching the GPT-3 work count as a falsification? if not, why not?
At a sufficiently high level of abstraction, I agree that “cost of experimenting” could be seen as the core difficulty. But at a very high level of abstraction, many other things could also be seen as the core difficulty, like “our inability to coordinate as a civilization” or “the power of intelligence” or “a lack of interpretability”, etc. Given this, John’s comment seemed like mainly rhetorical flourishing rather than a contentful claim about the structure of the difficult parts of the alignment problem.
Also, I think that “on our first try” thing isn’t a great framing, because there are always precursors (e.g. we landed a man on the moon “on our first try” but also had plenty of tries at something kinda similar). Then the question is how similar, and how relevant, the precursors are—something where I expect our differing attitudes about the value of empiricism to be the key crux.
Could you clarify what you mean by “the primary point” here? As in: the primary actual effect? Or the primary intended effect? From whose perspective?
RLHF helps with outer alignment because it leads to rewards which more accurately reflect human preferences than the hard-coded reward functions (including the classic specification gaming examples, but also intrinsic motivation functions like curiosity and empowerment) which are used to train agents in the absence of RLHF.
The smiley faces example feels confusing as a “classic” outer alignment problem because AGIs won’t be trained on a reward function anywhere near as limited as smiley faces. An alternative like “AGIs are trained on a reward function in which all behavior on a wide range of tasks is classified by humans as good or bad” feels more realistic, but also lacks the intuitive force of the smiley face example—it’s much less clear in this example why generalization will go badly, given the breadth of the data collected.
I take this comment as evidence that John would fail an intellectual turing test for people who have different views than he does about how valuable incremental empiricism is. I think this is an ITT which a lot of people in the broader LW cluster would fail. I think the basic mistake that’s being made here is failing to recognize that reality doesn’t grade on a curve when it comes to understanding the world—your arguments can be false even if nobody has refuted them. That’s particularly true when it comes to very high-level abstractions, like the ones this field is built around (and in particular the abstraction which it seems John is using, where making progress on outer alignment makes almost no difference to inner alignment).
Historically, the way that great scientists have gotten around this issue is by engaging very heavily with empirical data (like Darwin did) or else with strongly predictive theoretical frameworks (like Einstein did). Trying to do work which lacks either is a road with a lot of skulls on it. And that’s fine, this might be necessary, and so it’s good to have some people pushing in this direction, but it seems like a bunch of people around here don’t just ignore the skulls, they seem to lack any awareness that the absence of the key components by which scientific progress has basically ever been made is a red flag at all.
I think it’s possible to criticise work on RLHF while taking seriously the possibility that empirical work on our biggest models is necessary for solving alignment. But criticisms like this one seem to showcase a kind of blindspot. I’d be more charitable if people in the LW cluster had actually tried to write up the arguments for things like “why inner misalignment is so inevitable”. But in general people have put shockingly little effort into doing so, with almost nobody trying to tackle this rigorously. E.g. I was surprised when my debates with Eliezer involved him still using all the same intuition-pumps as he did in the sequences, because to me the obvious thing to do over the next decade is to flesh out the underlying mental models of the key issue, which would then allow you to find high-level intuition pumps that are both more persuasive and more trustworthy.
I’m more careful than John about throwing around aspersions on which people are “actually trying” to solve problems. But it sure seems to me that blithely trusting your own intuitions because you personally can’t imagine how they might be wrong is one way of not actually trying to solve hard problems.
There’s one attitude towards alignment techniques which is something like “do they prevent all catastrophic misalignment?” And there’s another which is more like “do they push out the frontier of how advanced our agents need to be before we get catastrophic misalignment?” I don’t think the former approach is very productive, because by that standard no work is ever useful. So I tend to focus on the latter, with the theory of victory being “push the frontier far enough that we get a virtuous cycle of automating alignment work”.
Some conceptual alignment research projects
Huh, I thought you agreed with statements like “if we had many shots at AI Alignment and could get reliable empirical feedback on whether an AI Alignment solution is working, AI Alignment would be much easier”.
I agree that having many shots is helpful, but lacking them is not the core difficulty (just as having many shots to launch a rocket doesn’t help you very much if you have no idea how rockets work).
I don’t really know what “reliable empirical feedback” means in this context—if you have sufficiently reliable feedback mechanisms, then you’ve solved most of the alignment problem. But, out of the things John listed:
Goodhart problems in outer alignment, deception in inner alignment, phase change in hard takeoff, “getting what you measure” in slow takeoff
I expect that we’ll observe a bunch of empirical examples of each of these things happening (except for the hard takeoff phase change), and not know how to fix them.
Alignment as a field is hard precisely because we do not expect to see empirical evidence before it is too late.
I don’t think this is the core reason that alignment is hard—even if we had access to a bunch of evidence about AGI misbehavior now, I think it’d still be hard to convert that into a solution for alignment. Nor do I believe we’ll see no empirical evidence of power-seeking behavior before it’s too late (and I think opinions amongst alignment researchers are pretty divided on this question).
No, I’m thinking of cases where Alice>Bob, and trying to gesture towards the distinction between “Bob knows that Alice believes X” and “Bob can use X to make predictions”.
For example, suppose that Bob is a mediocre physicist and Alice just invented general relativity. Bob knows that Alice believes that time and space are relative, but has no idea what that means. So when trying to make predictions about physical events, Bob should still use Newtonian physics, even when those calculations require assumptions that contradict Alice’s known beliefs.
Interesting post! Two quick comments:
Sometimes we analyze agents from a logically omniscient perspective. … However, this omniscient perspective eliminates Vingean agency from the picture.
Another example of this happening comes when thinking about utilitarian morality, which by default doesn’t treat other agents as moral actors (as I discuss here).
Bob has minimal use for attributing beliefs to Alice, because Bob doesn’t think Alice is mistaken about anything—the best he can do is to use his own beliefs as a proxy, and try to figure out what Alice will do based on that.
This makes sense when you think in terms of isolated beliefs, but less sense when you think in terms of overarching world-models/worldviews. Bob may know many specific facts about what Alice believes, but be unable to tie those together into a coherent worldview, or understand how they’re consistent with his other beliefs. So the best strategy for predicting when Bob is a bounded agent may be:
Maintain a model of Alice’s beliefs which contains the specific things Alice is known to believe, and use that to predict Alice’s actions in domains closely related to those beliefs.
For anything which isn’t directly implied by Alice’s known beliefs, use Bob’s own world-model to make predictions about what will achieve Alice’s goals.
I like your overall ambitions! I want to note a couple of things that seemed incongruous to me/things I’d change about your default plan.
I’m 24 now, so I’m hoping to start my career trajectory at 32 (8 years forms a natural/compelling Schelling point
This seems like very much the wrong mindset. You’re starting this trajectory now. In order to do great intellectual work, you should be aiming directly at the things you want to understand, and the topics you want to make progress on, as early as you can. A better alternative would be taking the mindset that your career will end in 8 years, and thinking about what you’d need to produce great work by that time. (This is deliberately provocative, and shouldn’t be taken fully literally, but I think points in the right direction, especially given that you’re aiming to do research where the credentials from a PhD that’s successful by mainstream standards don’t matter very much, like agent foundations research and more general high-level strategic thinking).
Pick a new important topic each month (or 2 −3 months)
Again, I’d suggest taking quite a different strategy here. In order to do really well at this, I think you don’t want the mindset of shallowly exploring other people’s work (although of course it’s useful to have that as background knowledge). I think you want to have the mindset of identifying the things which seem most important to you, pushing forward the frontier of knowledge on those topics, following threads which arise from doing so, and learning whatever you need as you go along. What it looks like to be successful here is noticing a bunch of ways in which other people seem like they’re missing stuff/overlooking things, digging into those, and finding new ways to understand these topics. (That’s true even if your only goal is to popularise existing ideas—in order to be able to popularise them really well, you want the level of knowledge such that, if there were big gaps in those ideas, then you’d notice them.) This is related to the previous point: don’t spend all this time preparing to do the thing—just do it!
I think that I am unusually positioned to be able to become such a person.
I think that doing well at this research is sufficiently heavy-tailed that it’s very hard to reason your way into thinking you’ll be great at it in advance. You’ll get far far more feedback on this point by starting to do the work now, getting a bunch of feedback, and iterating fast.
Good luck!
Thanks for this comment! I think it raises a very fair point. I do expect algorithmic improvements to make a significant difference—even if AGIs start off thinking more slowly than humans, algorithmic improvements would then allow us to train a smaller model with the same level of performance, but much faster. (This paper isn’t quite measuring the same thing, but “efficiency doubling every 16 months” seems like a reasonable baseline to me.) And techniques like model distillation would also help with that.
However, I think the claim as originally written was probably misleading, so I’ve rephrased it to focus on algorithmic improvements.
Another response is “The AI paralyzes your face into smiling.”
But this is actually a highly nontrivial claim about the internal balance of value and computation which this reinforcement schedule carves into the AI. Insofar as this response implies that an AI will primarily “care about” literally making you smile, that seems like a highly speculative and unsupported claim about the AI internalizing a single powerful decision-relevant criterion / shard of value, which also happens to be related to the way that humans conceive of the situation (i.e. someone is being made to smile).
Who do you think would make the claim that the AI in this scenario would care about “literally making you smile”, as opposed to some complex, non-human-comprehensible goal somewhat related to humans smiling? E.g. Yudkowsky gives the example of an AI in that situation learning to optimize for “tiny molecular smiley faces”, which is a much weirder generalization than “making you smile”, although I think still less weird than the goal he’d actually expect such a system to learn (which wouldn’t be describable in a single four-word phrase).
I think the AI will very probably have a spread of situationally-activated computations which steer its actions towards historical reward-correlates (e.g. if near a person, then tell a joke), and probably not singularly value e.g. making people smile or reward.
I think this happens when you have less intelligent systems, and then as you have more intelligent systems those correlates end up unified into higher-level abstractions which correspond to large-scale goals. I outline some of the arguments for that position in phase 3 here.
I intend to convert this report to a nicely-formatted PDF with academic-style references. Please comment below, or message me, if you’re interested in being paid to do this. EDIT: have now hired someone to do it.
More generally, I’ll likely make a number of edits over the coming weeks, so comments and feedback would be very welcome.
“AGI Safety Fundamentals: Let’s help people get an introduction to basic concepts in ML, AI, and AI safety.”
IMO this overstates how much ML we teach, there’s one week of very basic introduction to ML and then everything else is pretty directly alignment-related.