TLDR: In recent work, Roy Fox proposes to understand an agent’s capabilities in terms of the set of environment dynamics it can bring about.^[1] This leads to an intriguing duality between probabilities and utilities via the Legendre-Fenchel transform.

Introduction

Some agents are more powerful than others. Indeed, some can yield a wider range of outcomes, maybe because they are capable long-term planners or because they have built rich world models. Being able to clearly delineate the capabilities of agents is an important challenge for AI alignment.

A natural place to start thinking about how to describe the capabilities of an agent is reinforcement learning (RL), or more generally, approaches that see behaviour as arising from the maximisation of expected utility. By taking this view, one can describe “capability” as the range of reward/utility functions that an agent can successfully maximise — as done e.g. in classic work by Legg & Hutter and also in more recent work.

Such a perspective is very useful, but I am not a big fan of rewards/utilities. Rewards are great in games and other settings where they come naturally, but real life often does not handle rewards on a silver plate. When absent, rewards are often defined artificially — but the design of rewards that give rise to desired behaviour without enabling Goodharting is extremely difficult. That said, reward-based frameworks are powerful,^[2] and proposing alternative approaches that can compete with them is hard.^[3]

Thus, while I am not a utility/reward fan, I am intrigued by situations where they arise naturally. One place where this happens is in the celebrated von Neumann-Morgenstern utility theorem (VNM), which states that preferences with specific properties can be described as if the agent is trying to maximise a specific utility function (see this great post for related discussions). Another interesting idea is the link between utility maximisation and description length minimisation presented by John Wentworth, which suggests a duality between utilities and probabilities.^[4] A related perspective appears in Jeffrey–Bolker rotations as discussed by Abram Demski, where probability and value form vectors and certain linear transformations can move structure between them without changing the represented preferences.

I recently stumbled upon yet another way to see how utilities can naturally arise from probabilities, this time via the Legendre transform. In contrast with the settings discussed by VNM, Wentworth, and Demski, which assume a given preference or utility, here we simply start from achievable environmental dynamics to later derive utilities from them. The idea is simple and elegant — I’d not be surprised if it has been discussed before.^[5] Moreover, I believe this perspective may be useful for various issues related to AI alignment, which is what pushed me to write about it here.

The rest of this post presents:

A general way to describe the “capability space” of an agent. This description uses probabilities and policies, but doesn’t use utilities or rewards.
An explanation of how this definition affords a dual description via the Legendre transform, which naturally establishes utilities as Legendre duals of probabilities.

Defining capability space

Let’s consider an agent that acts over an environment described by a variable over a time-window from . For simplicity, let us focus on scenarios where and the actions that an agent can implement are finite.

Denote by the set of possible policies that a particular agent can feasibly implement, perhaps subject to information, computation, memory, or training constraints. In general, we assume that by acting according to a policy , the agent influences the dynamics of the environment. When considering a trajectory of environmental states , the effect of the policy is described by probabilities given by .

Example: Controlled Markov processes

To make this more concrete, let us denote the actions of the agent by , and assume that the dynamics of the environment are Markovian conditioned on its previous state and the agent’s action . Then, the probability of observing a given sequence of states and actions is

where is a Markov kernel. Then, we can define the dynamics of the environment when the agent acts according to simply as

In a recent paper, Roy Fox^[6] defines the capability space of an agent as the collection of environment stochastic dynamics that are achievable via the various policies that the agent could adopt. This definition comes with three assumptions:

The agent is free to choose any policy , and thus it can choose among environmental dynamics .
The agent can generate any mixture of realisable choices, so that is convex.
The agent can realise the limit of sequences of realisable choices, so that is closed.

One can say that an agent’s capability can be assessed via the size and shape of its capability space. This is a general definition that does not rely on utilities or rewards — instead, it uses a credal set as in imprecise probability.

The Legendre-Fenchel transform

Our next step will be to re-express the capability space using the Legendre Transform. But before doing that, let me provide an overview of what this transform is.^[7]

The Legendre-Fenchel transform was first introduced by Legendre in 1787, and was later extended by Fenchel in 1949. People are often more familiar with the Fourier transform, which takes a function in time domain and returns an alternative representation of it in frequency domain . Note that the Fourier transform provides two things: a dual variable ( instead of ) and a dual function ( instead of ). The Legendre-Fenchel transform operates in a similar manner: it takes a convex function and delivers two things: (i) a dual coordinate and (ii) a dual function expressed in those coordinates.

Concretely, for a given convex function defined over a convex set , the Legendre transform delivers a convex dual given by

where the domain of is . For strictly convex functions that are differentiable, the expression of the convex dual reduces to

where is the inverse of . The motivations behind this definition can be understood algebraically or geometrically, as discussed below.

Algebraic interpretation of the Legendre transform

Consider a smooth and strictly convex function . Because of its convexity, the derivative is an increasing function of . Because of this, the function establishes a natural bijection between and . Let’s use as a shorthand notation.

Due to the bijection, is just a re-labeling of ; one can think of and as alternative “coordinates” from which to express . One can indeed rewrite in terms of , creating a new function . If happens to be convex, then one can play the game again noting that establishes a bijection between and . By doing this, one could build further coordinates for .

But there is another interesting question one can consider: would it be possible to build a function such that the dual coordinate of goes back to being ?

The obvious candidate, , doesn’t work, as and are usually not the same. Luckily, a small tweak does the magic:

where is established using the bijection discussed above (but in the opposite direction). A quick calculation shows that just as . This construction is equivalent to the Legendre-Fenchel transform when is strictly convex and differentiable.

Geometric interpretation of the Legendre transform

Consider a smooth and strictly convex function . At each point , the tangent to has a unique slope . Similarly, the intercept is also unique. By denoting the value of the intercept for a given slope as , one finds that

Thus, is the Legendre transform of , and are the slopes and intercepts of the tangents of .

Interestingly, thanks to the symmetry of the equation above, the intercept of the tangent of for a given slope is again .

This derivation shows that the information that constitutes a strictly convex function can be equally conveyed by a collection of slope-intercept pairs. This is closely related to the fact that the epigraph of ,

is closed and convex if is lower-semicontinuous and convex, and closed convex sets can be expressed as intersections of half-spaces^[8]. Intuitively, this is like sculpting a block of marble into a convex shape by doing cuts of the block with a sharp knife.

For an epigraph, the above property takes the following form:

where is the set of slope-intercept pairs of the linear functions lying below the functions’ graph.

Under what conditions is the Legendre-Fenchel transform a “proper” conjugation — i.e., an involution? The Fenchel-Moreau theorem states that if is the Legendre-Fenchel dual of , then if is a proper lower-semicontinuous convex function.

Utilities as Legendre duals of probabilities

We are now ready for the central idea. Consider describing the capability space via an indicator function given by

if , and

if .

Note that is convex if is convex — thus, one can compute its Legendre-Fenchel transform. A quick calculation shows that the transform gives ^[9]

Thus, tells us the optimal performance that an agent with capability space can achieve on the task indexed by utility , being the convex dual of .

Moreover, the assumptions of convexity and closedness on make a proper lower-semicontinuous convex function; therefore, the Legendre-Fenchel transform of is again . This implies that all information in can be recovered from . Indeed, this alternative description leads to a dual characterisation of the capability space in terms of performance bounds:

This has various interesting implications: ^[10]

The Legendre-Fenchel transform states that the convex conjugate of the indicator function of the capability space is the maximal expected utility that such an agent can achieve.
The coordinates of the indicator function are probabilities; the dual coordinates of its convex conjugate are real-valued functions on trajectories, which formally look exactly like utility functions.

Conclusion

We have explored how the capability space of an agent can be defined in two alternative ways:

As the set of stochastic dynamics that the agent can elicit in its environment. Here, the capability is evaluated by providing a candidate distribution to , which then says if belongs to the possibilities of the agent or not.
As the range of expected utility values that the agent can achieve on a variety of tasks. Note that by highlighting the maximum, is delineating the possible values that are attainable. Here, capabilities are evaluated by testing various utility functions and checking how much expected utility the agent can attain.

What I find most fascinating is how utilities naturally emerge as duals of probability via the Legendre-Fenchel formalism. In this context, utilities don’t have the semantics that we normally attribute to them: they just highlight a dual domain that can be used to convey the same information regarding the dynamics of the environment that the agent can elicit. Mathematically, the duality is a consequence of the fact that convex sets of probabilities can be expressed as intersections of half-spaces, and utilities are a natural way to index such half-spaces.

Why this matters for alignment. A substantial portion of alignment is, in one way or another, about characterising what an agent could do rather than what it happens to do. The capability space turns comparing agents’ capabilities into set inclusion, and a safety property (“the agent cannot steer the world into region ”) becomes a geometric constraint on . On the dual side, the convex dual is exactly what evaluations measure: pick a task and see how well the agent does.

This suggests an interesting possibility: since determines completely, one could in principle reconstruct the full capability space — every dynamics the agent can elicit — purely from its performance on tasks. In practice, we can only evaluate finitely many tasks , and each evaluation yields a half-space

that must contain . Intersecting them gives a polytope that outer-approximates the capability space. For known parametric families of environments (e.g. finite MDPs with known transition structure), is a polytope with finitely many faces, so finitely many well-chosen tasks do recover it exactly.

^
I came across Roy’s work during the Finding the Frame workshop at the RL Conference. I strongly recommend this workshop to anyone interested in the foundations of reinforcement learning.
^
In the context of RL, I very much recommend the series of six talks on the RL Debate Series, which exhibit various arguments for and against modelling behaviour in terms of reward maximisation.
^
I really like empowerment and other approaches to intrinsic motivation, but it is still early days.
^
This link between probabilities and utilities has been generalised in this recent paper, about which I’ll write another post sometime soon.
^
If you have seen related ideas elsewhere, please let me know!
^
It would be great if Roy could write a blog here presenting this work.
^
Further discussions about the Legendre transform can be found in this paper and this paper.
^
See Theorem 11.5 in Rockafellar’s book.
^
Below, the sup turns into a max because is a closed subset of a probability simplex (which is compact).
^
Another, perhaps more anecdotal implication is that inherits the structure of the von Neumann–Morgenstern theorem for free. It is direct to see that for any and
.
That is, carries the same information for and for any positive affine transformation of it. In VNM, utilities being “unique up to positive affine transformations” is a consequence of axioms on preferences. Here, the same invariance falls out of the geometry: affine rescalings of don’t change which face of is selected, so they cannot be distinguished by what the agent can achieve. The dual coordinates are not just utility-like in name — they come with the equivalence classes that utilities have.