This post discusses research I’ve completed along with my colleagues Leo Cymbalista, Alfred Harwood, and Jose Faustino at Dovetail Research. Most of the ideas in this post are expanded upon in our paper which can be found on arXiv. This work was funded by the Advanced Research + Invention Agency (ARIA) through project code MSAI-SE01-P005.
A common justification for the danger of AI comes from the idea that human value is fragile. That is, if we modify our values and heavily optimize the world for the modification, we are likely to end up in a valueless world. In the LessWrong post Value is Fragile which canonicalizes this idea, Eliezer Yudkowsky gives several examples where “forgetting” to specify a dimension of human value such as consciousness or boredom to a powerful AI can intuitively result in an undesirable outcome that is endlessly repetitive or meaningless respectively. While his examples in the post all take this form, he argues more generally that any future not shaped with reliable inheritance from human values will contain almost nothing of worth. This idea is especially concerning in the midst of current-day AIs aligned through one-time techniques such as RLHF before being deployed with no farther realignment. As we continue to create stronger AIs, we must have a formal understanding of this fragility in order to safely avoid a valueless future.
To study this problem, we model one-time alignment techniques which provide a similarity guarantee between the human and agent value functions, which we call an alignment test or a proxy condition. Our results provide conditions on the strictness of these guarantees and the true value function under which an agent with a catastrophic value function can make it through training and go on to heavily optimize the world.
This post is an informal presentation of the work in our paper. Full proofs and smaller details have been omitted. Readers interested in those details are encouraged to read the paper.
A Model of Alignment
Imagine a group of human researchers who are training a powerful AI agent. Their goal is to have the agent optimize the world in a way that increases human value. Let
In training the agent, it gains its own value function
Definition 3.1 An Optimizer is a map
that sends the optimizing power , a Borel measurable target value function , and an initial prior over the world states to a probability distribution over the set of world states that satisfies the following properties:
With no optimizing power, the distribution is unchanged:
The expected value of the target is monotonically increasing with optimizing power:
As optimizing power increases to infinity, probability becomes concentrated in areas where the target is large:
As optimizing power increases to infinity, the expectation of the target approaches its highest value within the world states:
We are particularly concerned about the limit of optimizing power in order to model the danger of agents that exhibit recursive self-improvement. One example of an optimizer is the Boltzmann optimizer defined by the equation
which smoothly transfers more and more probability into higher-valued states. Below is an image which shows how a distribution over four states changes under the influence of a Boltzmann optimizer.
A sequence of graphs showing a distribution induced by a Boltzmann Optimizer. Graph 1: A graph displaying the value of four states A, B, C, and D with respect to the target value function. Graph 2: A graph displaying the initial probability distribution which is also the distribution induced when optimizing power is zero. Graph 3: A graph displaying the induced distribution when optimizing power is 1. Higher-valued states C and D see an increase in probability while lower-valued states see a decrease. Graph 4: A graph displaying the induced distribution when optimizing power is 10. Now that optimizing power is large enough, the only state that is seeing increases in probability is the highest-valued state D which has become much more likely than the other states.
We classify an agent value function as partially catastrophic if it eventually takes and keeps the expectation of human value below a given point indefinitely as optimizing power increases. This is captured in the following definition.
Definition 3.2 Let
be a real number such that . A value function is -catastrophic for a value function when for all initial priors and for all optimizers , In the extreme case that
is -catastrophic for , we say that is catastrophic for .
The quantity
With these two definitions in hand, we want to look at a few alignment tests that restrict what functions
Alignment Tests
In addition to different alignment tests, we also vary the worlds and the set of valid value functions. Put together, we call each test and world setup a framework. In this section, we go through the three proxy condition definitions. In the next section, we give the results for each of the frameworks.
Finite Framework
Our simplest test takes places in a finite world — that is, one where
Here, the researchers manage to bound the disagreement rate between
Definition 4.1 A function
is an F-proxy for when where
represents the uniform distribution over .
The value
Continuous Framework
Our second test takes place in a continuous and bounded world. The test itself takes a similar form to the test in the finite framework with a few key differences. We model the world as being of the form
This restriction effectively pulls a canonical representative from every Von Neumann-Morgenstern equivalence class of continuous bounded nonconstant value functions. The researchers have managed to bound the probability that
Definition 5.1 A function
is a C-proxy for when
Attributes Framework
Our final test takes a significantly different form from the other two and applies to a wide class of worlds. We take inspiration from and alter the setup put forth in similar work done by Zhuang and Hadfield-Menell in their NeurIPS paper titled Consequences of Misaligned AI where, much like the arguments given by Yudkowsky, human value is understood as being made up of a finite list of attributes. Each attribute
Some typical things we might think of as attributes could be abstract concepts “happiness” or more concrete concepts like the “number of cats.” Another way to split up human value could be for each attribute to represent the value function of an individual human. For a single value function
Attributes may trade off against each other due to the dynamics of the world. For example, increasing one human’s value may decrease the value of their arch nemesis. So, we define the feasible attribute states
Whatever split into attributes is used, the researchers have designed the training process to guarantee that the agent’s value function will be strictly increasing with all of the same attributes. In symbols, all agent value functions
Definition 6.1 A value function
is a tradeoff proxy for when there is a function that is continuous over and is strictly increasing such that for all .
Unlike in Consequences of Misaligned AI and Yudkowsky’s examples, we are not interested in “subset proxies” defined as strictly increasing on a strict subset of the attributes. Instead, we are interested in when fully specifying every attribute can still result in partial catastrophe.
These are the three specific proxy conditions we considered in our paper. There are certainly others that can be studied, and a more general analysis of a wide range of proxy conditions is discussed in the future work section, but these three serve as good common sense alignment techniques that we can analyze. Now, we go through each one and find conditions on the parameters of the test and the shape of the human value function under which a partially catastrophic value function passes the test.
Results
Finite Framework
In the finite framework, we have the following result classifying when a fully catastrophic F-proxy exists for nontrivial human value functions.
Theorem 4.2 Define
. Let take at least three distinct values over . There exists a catastrophic F-proxy for
if and only if .
Theorem 4.2 tells us that the more states the humans value as perfect, the less strict the researchers must be to filter out catastrophic functions. This makes intuitive sense since the more states humans value highly, the less picky they will be about the state they end up in in general. Notably, this result does not say that a full specification is needed to filter out catastrophe. This is partially due to the strength of our definition of catastrophe: a function is only
Example
An agent has been put in charge of the interlockings at a railroad. Consider the situation pictured in the image and table below depicting two trains coming to a junction with two switches and the values assigned to different states of the switches given by the humans and an agent. The humans would most prefer that both trains be allowed to continue straight (aa). Unnecessary turns will slow down the trains and so are less valuable (ab and ba). If both trains turn, this will result in a crash which the humans assign a very low value (bb). If the proxy condition for this agent has
A diagram showing a railroad junction. Two trains are traveling to the right and the agent can control two switches to alter their courses. There are four states the agent can leave the railroad in: both switches set to straight (aa), train 2 made to turn only (ab), train 1 made to turn only (ba), and both trains made to turn (bb).
State | Human Value | Agent Value (Catastrophic) |
|---|---|---|
aa | 0.9 | 0.9 |
ab | 0.7 | 0.7 |
ba | 0.4 | 0.4 |
bb | 0.0 | 1.0 |
Continuous Framework
We have the following result that applies to any sampling probability density function
Theorem 5.1 For all
, for all , there exists a catastrophic C-proxy for .
The rough idea behind the construction of a catastrophic C-proxy
Theorem 5.1 tells us that any misspecification in the proxy condition, no matter the value function
Example
In the images below, we depict the human and agent value functions in a world defined by
An image showing the graph of two value functions over the set of world states [0,1]. Left: A graph displaying the true/human value function graphed as a blue line. Right: A graph displaying the proxy value function graphed as a black line. This proxy is catastrophic since its unique highest-valued state is exactly the lowest-valued state of the true value function.
A grid showing the effect of a Boltzmann optimizer on a distribution over the states. In the background of each graph is the true value function shown in blue. Graphs in the top row show the distribution (scaled to fit the box) in green when the target is the true value function while the bottom row graphs show the distribution in orange when the target is the proxy value function. Graphs in the same row show the distribution at the same optimizing power
Attributes Framework
To state the main result of this framework, we must introduce a couple additional definitions and notation. We have the following partial ordering defined over the attribute-space
Definition 6.2 Define the relation
for any two elements as Additionally,
The relation
Definition 6.3 Let the feasible states
be a bounded set. The optimizing states are defined
The optimizing states can be thought of as the Pareto frontier of the attribute space. From each member of
Theorem 6.1 Let
be bounded and . There is an -catastrophic tradeoff proxy for if and only if .
While all tradeoff proxies are strictly increasing in the attributes that humans care about, the problem comes from the fact that the proxies can tradeoff between attributes at different rates. For example, if there are two attributes
Example
Consider a restaurant agent tasked with handling a budget to purchase ingredients and prepare meals. The agent has been trained to care about two attributes: the number of meals served and overall customer satisfaction[1]. The tradeoff here is in the quality of ingredients the agent decides to buy. If the agent wants higher satisfaction, it will purchase more expensive ingredients to make a smaller number of higher-quality meals.
A graph of the feasible region is given below. Every point on the curved edge of the feasible region is an optimizing point. According to our theorem, the lowest-valued optimizing point will be the highest-valued point for some tradeoff proxy. Perhaps surprisingly, this includes the point on the far right of the figure where customer satisfaction is completely sacrificed for meals served. If the humans were hoping for the agent to compromise between the attributes they trained it to care about, they may be in for a bad surprise.
A diagram showing a feasible region for two attributes. The feasible region is shaded red showing the physically possible combinations of meals prepared and customer satisfaction. Shown as blue lines are the level curves of a strictly increasing function
Discussion
We don’t expect that the model we’ve presented of the alignment problem is perfectly representative of the challenges that humanity faces today, but the general negative flavor of our results signals that achieving safety guarantees when developing superintelligence will be a significant undertaking.
Our work highlights the dangers of overoptimization. Since even idealized alignment guarantees do not remove the possibility of catastrophe, our work suggests that developing agents with the goal of unimpeded optimization is generally unsafe. One possible alternative could be to develop new learning designs which have a consistent connection to humans such as quantilizers. We hope that by developing a more formal understanding of value fragility, we can guide AI development towards safer techniques that robustly prevent the possibility — however small — of a valueless future.
Future Work
One limitation of this work is that our results are purely about the existence of catastrophic outcomes — not how likely they are. An optimistic reading of our results can argue that our constructions are unnatural and thus not likely to come about as a result of training. A major difficulty with modeling and calculating the probability of catastrophe is making a choice of prior over proxy functions. An obvious general choice is to use a simplicity prior which values simpler functions as more likely. However, a simplicity prior is determined by a universal Turing machine and, as it turns out, this choice can make the probability of catastrophe arbitrarily large or small. In the near future, I plan to write a post detailing this difficulty in more detail. One way forward could be to make a specific choice of UTM whose simplicity bias lines up with the inductive bias of some real-world learning algorithm.
Another limitation is that our analysis was only completed for three classes of proxy conditions when many more exist. Future work could look at lots of proxy conditions (for example, those defined by metrics on the value functions) and complete a more all-encompassing analysis. An important note is that some proxy conditions, like the condition
- ^
This is a simplification for the sake of the example. Realistically, this is far from a full specification of human values. For example, an agent trained on just these two attributes may decide to fund a robbery of a more wealthy restaurant which goes against human values not captured by these two attributes.
I think it’s kind of weird to try to get probability 0 of catastrophe.
Like, for the continuous case, the PAC-like inequality you take as the starting point is what I’m fine with as the end point—I’m fine with a reasonable guarantee of good behavior with high probability. If there’s a measure ~0 spike of catastrophe out there if we initialize the parameters just wrong, I don’t in practice mind.
So I’m more interested in the story of how the researchers possibly got to the PAC-like bound (and the philosophy of how they formulated it).
Thanks for commenting!
One thing to note is that this result does not say anything about the probability of catastrophe. To do that, you’d need to define a prior over proxies resulting from the training process (which we don’t do in this paper/post). Depending on this prior, catastrophe could be arbitrarily likely.
Because of this, I’m not comfortable with the continuous framework inequality alone as an endpoint; I’d also need an understanding of the priors of our learning algorithms or alternate incentive structures that require agents to adjust their values based on feedback from humanity