Former AI safety research engineer, now PhD student in philosophy of ML at Cambridge. I’m originally from New Zealand but have lived in the UK for 6 years, where I did my undergrad and masters degrees (in Computer Science, Philosophy, and Machine Learning). Blog: thinkingcomplete.blogspot.com
Richard_Ngo
Karma: 4,127
Great post, and I’m glad to see the argument outlined in this way. One big disagreement, though:
the Judge box will house a relatively simple algorithm written by humans
I expect that, in this scenario, the Judge box would house a neural network which is still pretty complicated, but which has been trained primarily to recognise patterns, and therefore doesn’t need “motivations” of its own.
This doesn’t rebut all your arguments for risk, but it does reframe them somewhat. I’d be curious to hear about how likely you think my version of the judge is, and why.
Thanks for the post; I think this type of reflection is very valuable. The main takeaway from this line of thought for me is that we’re in a community which selects for scrupulosity and caution as character traits, which then have a big impact on how we think about risks. This has various implications for thinking about AI, which I won’t get into here.
Thanks for the reply. To check that I understand your position, would you agree that solving outer alignment plus solving reward tampering would solve the pointers problem in the context of machine learning?
Broadly speaking, I think our disagreement here is closely related to one we’ve discussed before, about how much sense it makes to talk about outer alignment in isolation (and also about your definition of inner alignment), so I probably won’t pursue this further.
Above you say:
Now, the basic problem: our agent’s utility function is mostly a function of latent variables. … Those latent variables:
May not correspond to any particular variables in the AI’s world-model and/or the physical world
May not be estimated by the agent at all (because lazy evaluation)
May not be determined by the agent’s observed data
… and of course the agent’s model might just not be very good, in terms of predictive power.
And you also discuss how:
Human “values” are defined within the context of humans’ world-models, and don’t necessarily make any sense at all outside of the model.
My two concerns are as follows. Firstly, that the problems mentioned in these quotes above are quite different from the problem of constructing a feedback signal which points to a concept which we know an AI already possesses. Suppose that you meet an alien and you have a long conversation about the human concept of happiness, until you reach a shared understanding of the concept. In other words, you both agree on what “the referents of these pointers” are, and what “the real-world things (if any) to which they’re pointing” are? But let’s say that the alien still doesn’t care at all about human happiness. Would you say that we have a “pointer problem” with respect to this alien? If so, it’s a very different type of pointer problem than the one you have with respect to a child who believes in ghosts. I guess you could say that there are two different but related parts of the pointer problem? But in that case it seems valuable to distinguish more clearly between them.
My second concern is that requiring pointers to be sufficient to “to get the AI to do what we mean” means that they might differ wildly depending on the motivation system of that specific AI and the details of “what we mean”. For example, imagine if alien A is already be willing to obey any commands you give, as long as it understands them; alien B can be induced to do so via operant conditioning; alien C would only acquire human values via neurosurgery; alien D would only do so after millennia of artificial selection. So in the context of alien A, a precise english phrase is a sufficient pointer; for alien B, a few labeled examples qualifies as a pointer; for alien C, identifying a specific cluster of neurons (and how it’s related to surrounding neurons) serves as a pointer; for alien D, only a millennium of supervision is a sufficient pointer. And then these all might change when we’re talking about pointing to a different concept.
And so adding the requirement that a pointer can “get the AI to do what we mean” makes it seem to me like the thing we’re talking about is more like a whole alignment scheme than just a “pointer”.
The question then is, what would it mean for such an AI to pursue our values?
Why isn’t the answer just that the AI should:
1. Figure out what concepts we have;
2. Adjust those concepts in ways that we’d reflectively endorse;
3. Use those concepts?The idea that almost none of the things we care about could be adjusted to fit into a more accurate worldview seems like a very strongly skeptical hypothesis. Tables (or happiness) don’t need to be “real in a reductionist sense” for me to want more of them.
I agree with all the things you said. But you defined the pointer problem as: “what functions of what variables (if any) in the environment and/or another world-model correspond to the latent variables in the agent’s world-model?” In other words, how do we find the corresponding variables? I’ve given you an argument that the variables in an AGI’s world-model which correspond to the ones in your world-model can be found by expressing your concept in english sentences.
The problem of determining how to construct a feedback signal which refers to those variables, once we’ve found them, seems like a different problem. Perhaps I’d call it the “motivation problem”: given a function of variables in an agent’s world-model, how do you make that agent care about that function? This is a different problem in part because, when addressing it, we don’t need to worry about stuff like ghosts.
Using this terminology, it seems like the alignment problem reduces to the pointer problem plus the motivation problem.
I need some way to say what the values-relevant pieces of my world model are “pointing to” in the real world. I think this problem—the “pointers to values” problem, and the “pointers” problem more generally—is the primary conceptual barrier to alignment right now.
It seems likely that an AGI will understand very well what I mean when I use english words to describe things, and also what a more intelligent version of me with more coherent concepts would want those words to actually refer to. Why does this not imply that the pointers problem will be solved?
I agree that there’s something like what you’re describing which is important, but I don’t think your description pins it down.
I think ‘robust instrumentality’ is basically correct for optimal actions, because there’s no question of ‘emergence’: optimal actions just are.
If I were to put my objection another way: I usually interpret “robust” to mean something like “stable under perturbations”. But the perturbation of “change the environment, and then see what the new optimal policy is” is a rather unnatural one to think about; most ML people would more naturally think about perturbing an agent’s inputs, or its state, and seeing whether it still behaved instrumentally.
A more accurate description might be something like “ubiquitous instrumentality”? But this isn’t a very aesthetically pleasing name.
Can you elaborate? ‘Robust’ seems natural for talking about robustness to perturbation in the initial AI design (different objective functions, to the extent that that matters) and robustness against choice of environment.
The first ambiguity I dislike here is that you could either be describing the emergence of instrumentality as robust, or the trait of instrumentality as robust. It seems like you’re trying to do the former, but because “robust” modifies “instrumentality”, the latter is a more natural interpretation.
For example, if I said “life on earth is very robust”, the natural interpretation is: given that life exists on earth, it’ll be hard to wipe it out. Whereas an emergence-focused interpretation (like yours) would be: life would probably have emerged given a wide range of initial conditions on earth. But I imagine that very few people would interpret my original statement in that way.
The second ambiguity I dislike: even if we interpret “robust instrumentality” as the claim that “the emergence of instrumentality is robust”, this still doesn’t get us what we want. Bostrom’s claim is not just that instrumental reasoning usually emerges; it’s that specific instrumental goals usually emerge. But “instrumentality” is more naturally interpreted as the general tendency to do instrumental reasoning.
On switching costs: Bostrom has been very widely read, so changing one of his core terms will be much harder than changing a niche working handle like “optimisation daemon”, and would probably leave a whole bunch of people confused for quite a while. I do agree the original term is flawed though, and will keep an eye out for potential alternatives—I just don’t think robust instrumentality is clear enough to serve that role.
Yepp, this is a good point. I agree that there won’t be a sharp distinction, and that ML systems will continue to do online learning throughout deployment. Maybe I should edit the post to point this out. But three reasons why I think the training/deployment distinction is still underrated:
In addition to the clarifications from this post, I think there are a bunch of other concepts (in particular recursive self-improvement and reward hacking) which weren’t originally conceived in the context of modern ML, but which it’s very important to understand in the context of ML.
Most ML and safety research doesn’t yet take transfer learning very seriously; that is, it’s still in the paradigm where you train in (roughly) the environment that you measure performance on. Emphasising the difference between training and deployment helps address this. For example, I’ve pointed out in various places that there may be no clear concept of “good behaviour” during the vast majority of training, potentially undermining efforts to produce aligned reward functions during training.
It seems reasonable to expect that early AGIs will become generally intelligent before being deployed on real-world tasks; and that their goals will also be largely determined before deployment. And therefore, insofar as what we care about is giving them the right underying goals, then the relatively small amount of additional supervision they’ll gain during deployment isn’t a primary concern.
Nice post! A couple of quick comments:
“If interactions are repeated in an environment where the stakes get higher over time, most agents would prefer to be honest while the stakes are low, regardless of how they will act in a sufficiently high-stakes situation.”
If honesty in low-stakes situations is very weak evidence of honesty in high-stakes situations, then it will become less common as an instrumental strategy, which makes it stronger evidence, until it reaches equilibrium.
More generally, I am pretty curious about how reputational effects work when you have a very wide range of minds. The actual content of the signal can be quite arbitrary—e.g. it’s possible to imagine a world in which it’s commonly understood that lying continually about small scales is intended as a signal of the intention to be honest about large scales. Once that convention is in place, then it could be self-perpetuating.
This is a slightly extreme example but the general point remains: actions taken as signalling can be highly arbitrary (see runaway sexual selection for example) when they’re not underpinned by specific human mental traits (like the psychological difficulty of switching between honesty and lying).
“This holds especially because the higher the stakes get in a competition for expansion, the fewer future interactions one expects, as wiping out other players entirely becomes a possible outcome.”
Seems plausible, but note that early iterated interactions allows participant to steer towards possibilities where important outcomes are decided by many small interactions rather than a few large interactions, making long-term honesty more viable.
“lending an expensive camera to one’s sibling seems less risky than to a stranger simply because of the high likelihood of frequent future interactions”
This doesn’t seem right; your sibling is by default more aligned and trustworthy.
“while the agents can’t interpret each other or predict how well they would stick to commitments, a far more capable system (here, likely just a system with vastly more compute at its disposal) could do it for them.”
Is it fair to describe this as creating a singleton?
The burden of proof should be on whoever wants to claim that AI will be fine by default, not on whoever wants to claim it won’t be fine by default.
I’m happy to wrap up this conversation in general, but it’s worth noting before I do that I still strongly disagree with this comment. We’ve identified a couple of interesting facts about goals, like “unbounded large-scale final goals lead to convergent instrumental goals”, but we have nowhere near a good enough understanding of the space of goal-like behaviour to say that everything apart from a “very small region” will lead to disaster. This is circular reasoning from the premise that goals are by default unbounded and consequentialist to the conclusion that it’s very hard to get bounded or non-consequentialist goals. (It would be rendered non-circular by arguments about why coherence theorems about utility functions are so important, but there’s been a lot of criticism of those arguments and no responses so far.)
I agree with the two questions you’ve identified as the core issues, although I’d slightly rephrase the former. It’s hard to think about something being aligned indefinitely. But it seems like, if we have primarily used a given system for carrying out individual tasks, it would take quite a lot of misalignment for it to carry out a systematic plan to deceive us. So I’d rephrase the first option you mention as “feeling pretty confident that something that generalises from 1 week to 1 year won’t become misaligned enough to cause disasters”. This point seems more important than the second point (the nature of “be helpful” and how that’s a natural motivation for a mind), but I’ll discuss both.
I think the main disagreement about the former is over the relative strength of “results-based selection” versus “intentional design”. When I said above that “we design type 1 feedback so that resulting agents perform well on our true goals”, I was primarily talking about “design” as us reasoning about our agents, and the training process they undergo, not the process of running them for a long time and picking the ones that do best. The latter is a very weak force! Almost all of the optimisation done by humans comes from intentional design plus rapid trial and error (on the timeframe of days or weeks). Very little of the optimisation comes from long-term trial and error (on the timeframe of a year) - by necessity, because it’s just so slow.So, conditional on our agents generalising from “one week” to “one year”, we should expect that it’s because we somehow designed a training procedure that produces scalable alignment (or at least scalable non-misalignment), or because they’re deceptively aligned (as in your influence-seeking agents scenario), but not because long-term trial and error was responsible for steering us towards getting what we can measure.
Then there’s the second question, of whether “do things that look to a human like you’re achieving X” is a plausible generalisation. My intuitions on this question are very fuzzy, so I wouldn’t be surprised if they’re wrong. But, tentatively, here’s one argument. Consider a policy which receives instructions from a human, talks to the human to clarify the concepts involved, then gets rewarded and updated based on how well it carries out those instructions. From the policy’s perspective, the thing it interacts with, and which its actions are based on, is human instructions. Indeed, for most of the training process the policy plausibly won’t even have the concept of “reward” (in the same way that humans didn’t evolve a concept of fitness). But it will have this concept of human intentions, which is a very good proxy for reward. And so it seems much more natural for the policy’s goals to be formulated in terms of human intentions and desires, which are the observable quantities that it responds to; rather than human feedback, which is the unobservable quantity that it is optimised with respect to. (Rewards can be passed as observations to the policy, but I claim that it’s both safer and more useful if rewards are unobservable by the policy during training.)This argument is weakened by the fact that, when there’s a conflict between them (e.g. in cases where it’s possible to fool the humans), agents aiming to “look like you’re doing X” will receive more reward. But during most of training the agent won’t be very good at fooling humans, and so I am optimistic that its core motivations will still be more like “do what the human says” than “look like you’re doing what the human says”.
1. The goals that we imagine superintelligent AGI having, when spelled out in detail, have ALL so far been the sort that would very likely lead to existential catastrophe of the instrumental convergence variety.
2. We’ve even tried hard to imagine goals that aren’t of this sort, and so far we haven’t come up with anything. Things that seem promising, like “Place this strawberry on that plate, then do nothing else” actually don’t work when you unpack the details.
Okay, this is where we disagree. I think what “unpacking the details” actually gives you is something like: “We don’t know how to describe the goal ‘place this strawberry on that plate’ in the form of a simple utility function over states of the world which can be coded into a superintelligent expected utility maximiser in a safe way”. But this proves far too much, because I am a general intelligence, and I am perfectly capable of having the goal which you described above in a way that doesn’t lead to catastrophe—not because I’m aligned with humans, but because I’m able to have bounded goals. And I can very easily imagine an AGI having a bounded goal in the same way. I don’t know how to build a particular bounded goal into an AGI—but nobody knows how to code a simple utility function into an AGI either. So why privilege the latter type of goals over the former?
Also, in your previous comment, you give an old argument and say that, based on this, “they would have been justified in making the “vast majority of possible goals” claim”. But in the comment before that, you say “I disagree that we have no good justification for making the “vast majority” claim” in the present tense. Just to clarify: are you defending only the past tense claim, or also the present tense claim?
given the context of how people were thinking about things at the time, I feel like they would have been justified in making the “vast majority of possible goals” claim, even if they restricted themselves to more modest “wide range” claims.
Other people being wrong doesn’t provide justification for making very bold claims, so I don’t see why the context is relevant. If this is a matter of credit assignment, then I’m happy to say that making the classic arguments was very creditworthy and valuable. That doesn’t justify all subsequent inferences from them. In particular, a lack of good counterarguments at the time should not be taken as very strong evidence, since it often takes a while for good criticisms to emerge.
I disagree that we have no good justification for making the “vast majority” claim.
Can you point me to the sources which provide this justification? Your analogy seems to only be relevant conditional on this claim.
My point is that in the context in which the classic arguments appeared, they were useful evidence that updated people in the direction of “Huh AI could be really dangerous” and people were totally right to update in that direction on the basis of these arguments
They were right to update in that direction, but that doesn’t mean that they were right to update as far as they did. E.g. when Eliezer says that the default trajectory gives us approximately a zero percent chance of success, this is clearly going too far, given the evidence. But many people made comparably large updates.
Re counterfactual impact: the biggest shift came from talking to Nate at BAGI, after which I wrote this post on disentangling arguments about AI risk, in which I identified the “target loading problem”. This seems roughly equivalent to inner alignment, but was meant to avoid the difficulties of defining an “inner optimiser”. At some subsequent point I changed my mind and decided it was better to focus on inner optimisers—I think this was probably catalysed by your paper, or by conversations with Vlad which were downstream of the paper. I think the paper definitely gave me some better terminology for me to mentally latch onto, which helped steer my thoughts in more productive directions.
Re 2d robustness: this is a good point. So maybe we could say that the process orthogonality thesis is somewhat true, in a “spherical cow” sense. There are some interventions that only affect capabilities, or only alignment. And it’s sometimes useful to think of alignment as being all about the reward function, and capabilities as involving everything else. But as with all spherical cow models, this breaks down when you look at it closely—e.g. when you’re thinking about the “curriculum” which an agent needs to undergo to become generally intelligent. Does this seem reasonable?
Also, I think that many other people believe in the process orthogonality thesis to a greater extent than I do. So even if we don’t agree about how much it breaks down, if this is a convenient axis which points in roughly the direction on which we disagree, then I’d still be happy about that.
Ah, cool; I like the way you express it in the short form! I’ve been looking into the concept of structuralism in evolutionary biology, which is the belief that evolution is strongly guided by “structural design principles”. You might find the analogy interesting.
One quibble: in your comment on my previous post, you distinguished between optimal policies versus the policies that we’re actually likely to train. But this isn’t a component of my distinction—in both cases I’m talking about policies which actually arise from training. My point is that there are two different ways in which we might get “learned policies which pursue convergent instrumental subgoals”—they might do so for instrumental reasons, or for final reasons. (I guess this is what you had in mind, but wanted to clarify since I originally interpreted your comment as only talking about the optimality/practice distinction.)
On terminology, would you prefer the “training goal convergence thesis”? I think “robust” is just as misleading a term as “convergence”, in that neither are usually defined in terms of what happens when you train in many different environments. And so, given switching costs, I think it’s fine to keep talking about instrumental convergence.
Saying “vast majority” seems straightfowardly misleading. Bostrom just says “a wide range”; it’s a huge leap from there to “vast majority”, which we have no good justification for making. In particular, by doing so you’re dismissing bounded goals. And if you’re talking about a “state of ignorance” about AI, then you have little reason to override the priors we have from previous technological development, like “we build things that do what we want”.
On your analogy, see the last part of my reply to Adam below. The process of building things intrinsically picks out a small section of the space of possibilities.
Thanks for the feedback! Some responses:
This looks like off-line training to me. That’s not a problem per se, but it also means that you have an implicit hypothesis that the AGI will be model-based; otherwise, it would have trouble adapting its behavior after getting new information.
I don’t really know what “model-based” means in the context of AGI. Any sufficiently intelligent system will model the world somehow, even if it’s not trained in a way that distinguishes between a “model” and a “policy”. (E.g. humans weren’t.)
On the other hand, the instrumental convergence thesis basically says that for almost all goals, the AGI will have the specific convergent instrumental subgoals. If this is true, then this definitely applies to minds trained through ML, as long as their goals fall into the broad category of the thesis. So this thesis is way more potent for trained minds.
I’ll steal Ben Garfinkel’s response to this. Suppose I said that “almost all possible ways you might put together a car don’t have a steering wheel”. Even if this is true, it tells us very little about what the cars we actually build might look like, because the process of building things picks out a small subset of all possibilities. (
Also, note that the instrumental convergence thesis doesn’t say “almost all goals”, just a “wide range” of them.Edit: oops, this was wrong; although the statement of the thesis given by Bostrom doesn’t say that, he says “almost all” in the previous paragrah.)
Nice post. The one thing I’m confused about is:
It seems to me that the type of research you’re discussing here is already seen as a standard way to make progress on the full alignment problem—e.g. the Stiennon et al. paper you cited, plus earlier work on reward modeling by Christiano, Leike, and others. Can you explain why you’re institutionally uncertain whether to prioritise it—is it because of the objections you outlined? But your responses to them seem persuasive to me—and more generally, the objections don’t seem to address the fact that a bunch of people who are trying to solve long-term alignment problems actually ended up doing this research. So I’d be interested to hear elaborations and defences of those objections from people who find them compelling.