Definitions of “objective” should be Probable and Predictive

Rohin Shah6 Jan 2023 15:40 UTC
LW: 43 AF: 28
27 comments12 min readLW link

Introduction

Core arguments about existential risk from AI misalignment often reason about AI “objectives” to make claims about how they will behave in novel situations. I often find these arguments plausible but not rock solid because it doesn’t seem like there is a notion of “objective” that makes the argument clearly valid.

Two examples of these core arguments:

  1. AI risk from power-seeking. This is often some variant of “because the AI system is pursuing an undesired objective, it will seek power in order to accomplish its goal, which causes human extinction”. For example, “The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.” This is a prediction about a novel situation, since “causing human extinction” is something that only happens at most once.

  2. AI optimism. This is often some variant of “we will use human feedback to train the AI system to help humans, and so it will learn to pursue the objective of helping humans.” Implicitly, this is a prediction about what AI systems do in novel situations; for example, it is a prediction that once the AI system has enough power to take over the world, it will continue to help humans rather than execute a treacherous turn.

When we imagine powerful AI systems built out of large neural networks[1], I’m often somewhat skeptical of these arguments, because I don’t see a notion of “objective” that can be confidently claimed is:

  1. Probable: there is a good argument that the systems we build will have an “objective”, and

  2. Predictive: If I know that a system has an “objective”, and I know its behavior on a limited set of training data, I can predict significant aspects of the system’s behavior in novel situations (e.g. whether it will execute a treacherous turn once it has the ability to do so successfully).

Note that in both cases, I find the stories plausible, but they do not seem strong enough to warrant confidence, because of the lack of a notion of “objective” with these two properties[2]. In the case of AI risk, this is sufficient to justify “people should be working on AI alignment”; I don’t think it is sufficient to justify “if we don’t work on AI alignment we’re doomed”.

The core difficulty is that we do not currently understand deep learning well enough to predict how future systems will generalize to novel circumstances[3]. So, when choosing a notion of “objective”, you either get to choose a notion that we currently expect to hold true of future deep learning systems (Probable), or you get to choose a notion that would allow you to predict behavior in novel situations (Predictive), but not both.

This post is split into two parts. In the first part, I’ll briefly gesture at arguments that make predictions about generalization behavior directly (i.e. without reference to “objectives”), and why they don’t make me confident about how future systems will generalize. In the second part, I’ll demonstrate how various notions of “objective” don’t seem simultaneously Probable and Predictive.

Part 1: We can’t currently confidently predict how future systems will generalize

Note that this is about what we can currently say about future generalization. I would not be shocked if in the future we could confidently predict how the future AGI systems will generalize.

My core reasons for believing that predicting generalization is hard are that:

  1. We can’t predict how current systems will generalize to novel situations (of similar novelty to the situations that would be encountered when deliberately causing an existential catastrophe)

  2. There are a ridiculously huge number of possible programs, including a huge number of possible programs that are consistent with a given training dataset; it seems like we need strong evidence to narrow down the space enough that we can make predictions about generalization.

These are not decisive; it is simply an uninformative prior from which I start. It is also not necessarily hard to get strong evidence. For example, I am happy to confidently predict that given an English sentence that it has never seen before, GPT-3 would continue it with more English[4]. But I haven’t seen arguments that persuade me to be confident about how future systems will generalize. I’ll go through some of them below.

(Note that, while many of these arguments are inspired from things I’ve read or heard, the presentation here is my own and may not accurately represent anyone else’s beliefs.)

Laserlike plans /​ core of general intelligence. One argument is that if we assume that future deep learning systems are capable of e.g. building nanosystems, then they must be performing coherent consequentialist cognition, which allows us to predict some aspects of how they would generalize. In particular, while we can’t predict what goal they will pursue, we can predict that they will seek resources and power and manipulate or destroy humans in order to achieve the goal.

You can also make a stronger claim as follows. Most powerful cognition arises from core simple patterns underlying intelligence, such as getting stuff that allows you to do more stuff in the future, taking decisions based on whether it creates more of the stuff you want, etc. The first powerful AGI systems will use these patterns, simply because it is very difficult to get powerful AGI systems that don’t use these patterns, given how simple and useful they are. This argument is similar to the previous argument, but makes a stronger claim that we get a specific simple core algorithm.

There is a lot of discussion about this point and I won’t get into it here, but suffice it to say that I don’t have high confidence in this story.

General-purpose search. This argument says that because general-purpose retargetable search is so useful, that is how our AI systems will work; once you know you have a search algorithm then the standard argument of convergent instrumental subgoals applies.

My current belief is that this is a plausible way that future AI systems could work, but it’s just one of many possible architectures and not one that I am confident will arise. (See also this comment chain.)

Strong selection. This argument says that gradient descent will work very well, and so functions that score higher on the loss function will be much more likely than those that score lower. An AI system that directly cares about getting low loss will likely get lower loss than one that cares about doing what we want, and so we are likely to get one that cares directly about getting low loss (which in turn implies misaligned power-seeking).

My worry with this argument is that, while I would feel pretty confident in this argument in the limit of “max SGD capabilities”, it’s not obvious that it applies to the first superhuman AI systems that we build. Such systems are not going to be anywhere near the literally optimal performance on “getting low loss”; it seems like an open question whether getting to superhuman level requires “directly caring about loss” rather than some other internal reasoning architecture.

Conceptual clarity. This argument states that any powerful AI system must have clear concepts, that is, concepts which work well for a wide variety of tasks (at the very least, the training tasks), and which should thus be expected to work well in novel situations too. For a specific version of this argument, see Alignment by Default[5].

I certainly agree that this allows you to make some confident predictions about generalization behavior. For example, I expect GPT-3 has conceptual clarity about spelling and grammar. Even in most novel situations, as long as we start with good spelling and grammar, I predict it will continue to produce text with good spelling and grammar.

However, just knowing that the AI system has good concepts doesn’t tell you much about how it will use these concepts. An AI system that has a robust concept of manipulation could use it to protect you from propaganda, or to persuade you to give it more autonomy with which to pursue its own goals. It need not help to see what the system does during training: just because it was helpful during training and it has clear concepts doesn’t mean that it isn’t biding its time until it can execute a treacherous turn.

Simplicity bias. This argument says that deep learning has a simplicity bias; by reasoning about what algorithms are simple we can predict the generalization of future deep learning systems.

As with previous arguments, I think simplicity bias allows you to make predictions like “the AI won’t set money on fire”, “the AI won’t believe that 2 + 2 = 5”, “GPT-3 will continue to have good spelling and grammar”, and so on. (These predictions need not hold for a giant lookup table; we rule lookup tables out because of simplicity.) However, I don’t see how you argue for the AI risk or AI optimism stories, except by using simplicity bias to argue for one of the more specific arguments above.

Human analogy. This argument says that we can predict how humans can generalize, and a trained deep learning system is quite analogous to a human, and so we will be able to predict a trained deep learning system using the same techniques.

There are several different responses to this argument, including but not limited to:

  1. Humans use an input/​output space that we are very familiar with, making them easier to predict.

  2. The default guess that other humans behave similarly to how we would behave works reasonably often, but would not work as well for AI systems, since they reason in an alien manner.

  3. We’re not actually very good at predicting how humans will behave in unusual situations.

Short horizons. This argument suggests that AIs will only care about completing tasks with relatively short horizons, because that’s what they were trained on. As a result, we can predict that they would not pursue convergent instrumental subgoals.

I don’t find this persuasive because of the possibility of goal misgeneralization. For example, our short horizon tasks will be chosen to optimize for long horizon outcomes (e.g. a CEOs day to day tasks are meant to lead to long-term company success), and so the AI system may end up caring directly about long horizon outcomes.

Part 2: There are many types of objectives; none are both Probable and Predictive

In this section I’ll argue that there isn’t a notion of “objective” that is Probable and Predictive.

The core argument is just the one I laid out in the introduction: to have a notion of ‘objective’ that is Probable and Predictive, we would need to know how future systems would generalize to novel situations, but we don’t currently know this. But as further support and to give a better sense of where I’m coming from, I’ll also list out a few different notions of “objective” and show how they fail at least one of the two criteria.

I see definitions of “objectives” as varying along one key axis: how behavioral or structural the definition is. A structural definition identifies some object as the “objective”, and argues that it drives the agent’s behavior. In contrast, a behavioral definition looks at the agent’s behavior, and infers the “objective” from that behavior. As a simple example, the VNM theorem constructs a utility function (objective) out of preferences over lotteries (behavior); such a utility function is thus a behavioral objective.

Structural objectives

We’ll consider two types of structural objectives: outer and inner structural objectives.

Structural (outer): Here, the “objective” is identified with a particular part of the training process; for example, in deep RL it would be the reward function. I think such objectives are not Predictive. Current AI systems trained with a particular reward function do not generalize to continue to pursue that reward function in novel situations. Typically they just break, though goal misgeneralization gives specific examples in which they generalize competently to a different objective. It is an open question (to me) whether future systems will generalize to pursue the reward function used during training.

You can also see that the concept is problematic through other observations:

  1. This concept can vary wildly in its predictions for very similar systems. For example, we could incentivize exploration either by adding a novelty-seeking term to the reward, or by changing the action selection mechanism to bias towards actions that produce the most disagreement in an ensemble of dynamics models. These two mechanisms have similar effects on agent behavior, but wildly different outer structural objectives; this seems worrisome.

  2. Related to the previous point, sometimes it is hard to tell what the “objective” is in a particular agent implementation – what if there is logic that is separate from the gradient-based optimization? (Such as a safety shield that prevents the agent from taking certain actions in certain situations.)

  3. The only aspect of the outer structural objective that matters is its values on the training data. You could hypothetically “change” the values of the outer structural objective for non-training inputs, but the agent would be completely unaffected. So the outer structural objective is only relevant up to its values on the training data, and its values outside the training data do not matter. (This also applies to online learning setups, where “training data” now means “all the data seen in the past”.) If I can vary the outer structural objective significantly without changing the trained AI system at all, the outer structural objective is unlikely to be Predictive.

  4. We can train AI systems with a reward function, and then deploy the AI system without the reward function, and everyone expects this to work normally rather than e.g. the AI system doing everything it can to get the humans to reinstate the reward function at deployment.

For a more mechanistic treatment, see Reward is not the optimization target.

Structural (inner): This version of an objective requires an assumption of the form, “the model weights implement some form of mechanistic search or optimization”. The inner structural objective is then identified as the metric used to guide this search /​ optimization. We might classify this assumption into two forms:

  1. Strict interpretation: The model is a giant circuit that considers a wide variety of actions or plans, predicts their long-term outcomes accurately, evaluates the outcomes using a metric, and then executes the action that scores highest. We identify the metric as the “objective”.

    1. Under this interpretation, it seems like such objectives are not Probable: I don’t see why we should confidently expect neural nets to implement such a procedure.

    2. This isn’t the only possible strict interpretation. For example, you could also tell a story about how the model backchains by reasoning about what subgoals help towards a final goal, and consider that “final goal” to be the inner structural objective. But I still have the same objection, that such objectives do not seem Probable.

  2. Loose interpretation: The model is performing something vaguely like optimization towards some goal, and we can mostly guess what the goal is based on its behavior in the situations we’ve seen.

    1. In this case, it doesn’t seem like the argument can constrain my expectations enough for me to have predictions about the agent’s behavior in novel circumstances, and so such objectives are not Predictive.

I could imagine that some interpretation that is in between these two could be both Probable and Predictive, but I don’t currently see how to do it (and I don’t think anyone else has suggested a way to do it that I would find compelling).

You might try to rescue the strict interpretation by arguing that deep learning has a simplicity bias and the circuit described in the strict interpretation is the most “simple”, thus making it very Probable. However, I don’t think this works. Consider an agent with lots of real-world knowledge that was finetuned to solve simply connected mazes during training. It seems like you could get any of the following, all of which seem quite simple:

  1. An agent that follows the wall follower algorithm.

  2. An agent that builds an abstract model of the maze, and then runs depth first search to solve the maze.

  3. An agent that “wants” to maximize the number in the memory cell that corresponded to reward during training.

  4. An agent that “wants” to make paperclips (that knows that it would be shut down if it didn’t solve mazes now).

Behavioral objectives

If I see that AlphaZero tends to take moves that lead it to win at Go, it makes sense to say that its objective is to win at Go, even if it isn’t literally optimal at playing Go. However, in the general case, this sort of concept only makes sense on the set of inputs where you originally observed the behavior, in which case it doesn’t necessarily help you predict behavior in novel circumstances.

We’ll again consider two types of behavioral objectives: everywhere-behavioral and training-behavioral objectives.

Behavioral (everywhere): Here, the “objective” is a function U such that the agent’s behavior can be described as maximizing U, not just in the training distribution but in all possible situations that could arise (except for “unfair” situations, e.g. situations in which an adversary completely rewrites the weights of the AI system). This faces a lot of theoretical problems:

  1. It’s hard to apply this to humans. I might be able to say something like “currently, Alice’s goal is to relieve her hunger” (e.g. if she’s making a sandwich), but it seems much harder to say anything about Alice’s overall life objective, the thing that all of her actions are driving towards. (And even Alice probably can’t tell you what her overall life objective is, in a way that lets you actually predict what she will do in the future.)

  2. To the extent we could apply it to humans, it seems like we’d get an answer that is underdefined and changes over time.

  3. I suspect that you will often get a vacuous encoding of the policy (along the lines of the construction in this post).

  4. Even in theory we don’t know how to distinguish between biases and objectives.

If you count vacuous encodings of the policy as everywhere-behavioral objectives, then they aren’t Predictive: there’s no way to use knowledge of the training data to predict behavior in novel circumstances.

If you require the everywhere-behavioral objective to be “simple” (i.e. something like “maximize paperclips”), then they aren’t Probable: I don’t see a strong argument that deep learning systems must have such objectives.

Behavioral (training): Here, the “objective” is identified as a function U such that the agent’s behavior on the training distribution can be explained as maximizing U. The core problem with this definition is that there are lots of possible U’s that are consistent with the behavior on the training distribution, that make different predictions outside of the training distribution. As a result, this notion of “objective” can’t make predictions in novel circumstances, and so is not Predictive.

You might try to rescue this approach by taking the simplest U that explains the training behavior and arguing that deep learning has a simplicity bias, but this still doesn’t work, for the same reason that it didn’t work for strict inner structural objectives.

Summary

It seems quite hard to get a notion of “objective” that is both Probable and Predictive – the attempts I’ve made here don’t work.

Type of objectiveInterpretationProbablePredictive
Structural (outer) YesNo
Structural (inner)Strict: giant circuit evaluates outcomes using a metricNoYes
Structural (inner)Loose: performs something like optimization towards some goalYesNo
Behavioral (everywhere)Vacuous encodings of the policy countYesNo
Behavioral (everywhere)Require objective to be simpleNoYes
Behavioral (training) YesNo

Personally, I’m inclined to avoid trying to say that an AI “has an objective”, and instead talk directly about generalization behavior in novel situations. For example, I would suggest saying things like “in training situations the AI has tended to do X; in test situation Y I expect it to generalize to show behavior Z because of reason R”. This is usually what you’re using the word “objective” for anyway; this just forces you to spell out the inference that you are making. The arguments in Part 1 are examples of what this could look like.

Another approach would be to search for an improved notion of an “objective” that is both Probable and Predictive, and use that notion of “objective” in our arguments. I view the work on goal-directedness as aiming for this goal.

  1. ^

    The restriction to deep learning is important. For example, if you somehow ran AIXI, I feel relatively confident that you get misaligned pursuit of convergent instrumental subgoals, either from the search for optimal actions finding actions that take control of the reward-generating process, or from some other agents manipulating AIXI’s predictions in order to take control themselves (see this post).

  2. ^

    People familiar with my beliefs might be confused here, since I am generally in support of building an AI system that is always “trying” to do what we want. Isn’t this just a different way of saying that the AI system has an objective of doing what we want? I have two responses here.

    The more important response is that I only use “trying” to define the goal to which we aspire: I don’t use the concept to make strong claims about the extent to which we succeed at our goal. It seems quite plausible to me that we don’t succeed at the goal because the notion of “always trying to do X” is not sufficiently Probable. Note that lack of success does not imply that an existential catastrophe has occurred. An AI system that occasionally avoids asking clarifying questions that it knew it should have asked is not “trying to do what we want”, but that doesn’t mean it causes an existential catastrophe.

    The less important response is that, in AI safety, when people say “objective”, they want a much thicker concept than just “what the agent is trying to do”. They seem to want a concept from which you can derive “the AI will kill us unless we get the objective exactly right”. I don’t think you get these sorts of conclusions if you just talk about “trying” in its normal English-language meaning. For example, I can reasonably say that Bob is “trying” to win a game of chess, without implying that he wants to convert the universe into computronium for the purpose of solving chess to guarantee that he wins the game.

  3. ^

    A lot of the argumentation in this post depends on the concept of “novel situations”, but it is not totally clear what this means. The most expansive definition would define it as “any input not present in the training dataset”, but this is too broad a definition. GPT-3 may never have seen “The ocean is filled with saltwater creatures that are too small to be seen by the naked eye” during training, but it is similar enough that you can expect GPT-3 to generalize to that sentence. In contrast, a situation in which GPT-3 is asked to complete a sentence in a newly-discovered ancient language would clearly be a “novel situation”.

    The actual situation is more complicated; at the very least you’d want to view novelty as a spectrum and talk about how novel a situation is. For the purpose of this post, I will mostly ignore this. Whenever I talk about “novel situations”, you should be thinking of situations that are as novel as the situations that would occur if an AI deliberately enacts a plan leading to an existential catastrophe.

  4. ^

    With some exceptions, e.g. sentences like the “The translation of ‘table’ to French is ____”. I expect many of the examples in this post have these sorts of “uninteresting” exceptions; I’m not going to point out future instances.

  5. ^

    Note that the post assigns only 10% chance of the suggested path working in the short term and 5% in the long term, so it is consistent with my belief that the arguments can suggest plausibility but not confidence.

Rohin Shah6 Jan 2023 15:40 UTC
LW: 43 AF: 28
27 comments12 min readLW link