Natural Abstraction: Convergent Preferences Over Information Structures

[paulom]

The natural abstraction hypothesis claims (in part) that a wide variety of agents will learn to use the same abstractions and concepts to reason about the world.

What’s the simplest possible setting where we can state something like this formally? Under what conditions is it true?

One necessary condition to say that agents use ‘the same abstractions’ is that their decisions depend on the same coarse-grained information about the environment. Motivated by this, we consider a setting where an agent needs to decide which information to pass though an information bottleneck, and ask: when do different utility functions prefer to preserve the same information?

As we develop this setting, we:

• unify and extend the Blackwell Informativeness and Good(er) Regulator theorems

• investigate the minimum structure needed for strong natural abstractions (in this sense) to exist

• give examples where strong natural abstractions a) exist b) don’t exist c) exist at some levels of capability, but not at others—including cases where the natural abstractions at different sizes are orthogonal

• demonstrate that (very) weak natural abstractions exist under weak conditions

• formally connect natural abstractions to instrumental power-seeking incentives, and characterize them in a way inspired by Turner’s power-seeking theorems.

(As far as I can tell, apart from the obvious cited Theorems, these results are original^[1])

Whereas Wentworth’s abstraction work has focused on what natural abstractions might look like, and why we might expect them to exist, this work focuses on exploring what it would mean for something to be a natural abstraction in the first place. Rather than an attempt to answer the natural abstraction hypothesis, it is more an attempt to understand the question.

1. Base setting

Any agent exists within an environment that is bigger than itself. It therefore necessarily maps a large number of potential environment states to a smaller number of internal states. When it then chooses what action to take, this choice depends only on its internal state.

Roughly, we can consider this mapping from the environment to action-influencing internal states to be the agent’s abstractions^[2]. We want to understand when different utility functions will share preferences about how to do this mapping—under the natural abstraction hypothesis, in appropriate conditions, there should be certain abstractions that are convergent across most utility functions.

To model this, we’ll consider a game where:

1. There is some initial context

2. An agent writes a message, conditional on this context

3. The agent takes an action, conditional on the message

4. The agent gets some utility that depends on the context and its action.

We’ll keep everything finite to start—any infinities we want we should come by honestly, by taking limits. Our one concession to this is that we’ll allow normal real-valued probabilities and utilities (though we’ll see eventually how we could do without if we felt like it).

Formally, define:
$X$: a set of context states $M$: a set of signals
$A$: a set of actions
$U$: $(X,A)\to R$, a utility function
$\phi$: $X\to \Delta M$, an encoder mapping states to a distribution of signals
$\pi$: $M\to \Delta A$, a policy mapping signals to a distribution over actions

We allow $\phi$ and $\pi$ to be stochastic (stochastic maps), but we’ll also pay special attention to the case where they are deterministic (so effectively $\phi :X\to M$, $\pi :M\to A$), because it is especially clean and factors apart some of the dynamics we’re interested in.

Now, we are interested in preferences over $\phi$ assuming that $\pi$ is chosen optimally. When are these preferences, and the optimal encoders, the same for different $U$? What happens as we vary $\left|M\right|$? For the time being, we can equally consider different $U$ to be different agents or different tasks faced by the same agent. Later, we’ll see some places where the two interpretations diverge.

As one might expect of such a simple setup, similar formalisms have been considered in many other contexts. Most directly, exactly this game is studied in the decision theory and economics literature—with partial overlap with our questions. There, the objects $(\phi ,M)$ are commonly referred to as ‘information structures’ or ‘Blackwell experiments’; we’ll use the former term.

One other remark on notation: we’ll often be interested in comparing different encoders $\phi ,{\phi }^{\prime }$, which may have different signal spaces $M_{\phi },M_{{\phi }^{\prime }}$. We’ll write $\left|\phi \right|$ for $\left|M_{\phi }\right|$, the cardinality of the signal set.

1.1 Equivalence classes of encoders

One thing to note out of the gate: ‘the same encoder’ here shouldn’t mean equality as functions. For our current purposes, we care only about their information content—so if $\phi ,{\phi }^{\prime }$ are the same except for a relabelling of $M$, we want to consider them identical (and they will certainly have the same attainable utility for any $U$). Another way to say this: we want to quotient by $\mathrm{Aut}\left(M\right)=S_{\left|M\right|}$, i.e. the permutations of $M$, to get our real objects of interest. What are these objects?

In the deterministic case, they are the distinct possible sets of preimages $\left\{{\phi }^{-1}\right(m):m\in M\}$ - which are just the $\left|M\right|$-part partitions of $X$. In the general case, they are a more general partition-like object. Note that we can represent a stochastic map $\phi$ as an $\left|M\right|\times \left|X\right|$ column-stochastic matrix: a matrix with elements in $[0,1]$ such that the sum of each column is $1$: ${\sum }_{i=1}^{\left|M\right|}[\phi {]}_{ij}=1$. Then $[\phi {]}_{ij}$ gives the probability that we encode signal $i$ given we are in state $j$ - that is, $p_{\phi }\left(m_i|s_j\right)$. If we are indifferent to permutations of $M$, then we are indifferent to permutations of the rows, so we have a set (rather than vector) of row vectors. (If $X$ were instead a suitable topological space and $\phi$ continuous, then these would be exactly partitions of unity)

Sometimes it will be more convenient to think of $\phi$ as a function, sometimes to think of it as a partition $P_{\phi }$, and sometimes as a matrix; we’ll switch between all three as appropriate. However, keep in mind we really only care about $\phi$ up to equivalence in this sense.

1.2 Comparison of encoders

For a given utility function $U$, we take the value of an information structure to be the expected utility if we have that information structure and follow the optimal policy.

Formally, define $U\left(\phi \right)={max}_{\pi }{E}_{X,\pi \left(\phi \right(X\left)\right)}[U(x,a)]={max}_{\pi }{\sum }_{X,A}[\pi \circ \phi {]}_{xa}U(x,a)$; we can also say it is ${max}_{\pi }U((X,(\pi \circ \phi \left)\right(X\left)\right))$, extending $U$ to a function on distributions by taking the expectation. (By convention we’re assuming a uniform prior on $X$; this choice turns out to be irrelevant for our purposes).

We say that $U$ prefers $\phi$ to ${\phi }^{\prime }$ if $U\left(\phi \right)\ge U\left({\phi }^{\prime }\right)$, which we’ll also write $\phi {\ge }_U{\phi }^{\prime }$.

2. Basic properties for individual $U$

As a first step, to get a feel for the setting, let’s start with some observations about what can happen for arbitrary individual $U$. These give us one bound on common properties—for something happen for ‘many’ $U$, it must certainly happen for some particular $U$.

2.0 $M$ should be a bottleneck

One basic observation is that the problem becomes trivial if $\left|M\right|\ge \left|X\right|$ or $\left|M\right|\ge \left|A\right|$. At that point, we can always pick the globally optimal action for each value of $X$. For this setting to be remotely interesting, $M$ needs to be an information bottleneck.

2.1a Any deterministic partition is optimal for some $U$

This is also easy to see: for a partition $P=\{p_1,...,p_k\}$, take any $\{a_1,...,a_k\}\subseteq A$ and define $U(x,a_i)=1(x\in p_i)$. Furthermore, any optimal partition for this $U$ must be a refinement of $P$, and $P$ is the unique optimal partition with $k$ parts.

2.1b For any $U$, any nondeterministic $\phi$ is weakly dominated by some deterministic ${\phi }^{\prime }$

It is not quite literally the case that no nondeterministic $\phi$ is optimal for its size for any $U$. If $U$ is indifferent between where to put $x$ in the optimal policy, say $U(x,{\pi }_{opt}(m_1\left)\right)=U(x,{\pi }_{opt}(m_2\left)\right)$, then $x$ can be freely sent to any mixture of $m_1$ and $m_2$. But there is always a deterministic ${\phi }^{\prime }$ of less than or equal size which does at least as well, and ${\phi }^{\prime }$ either does strictly better or is strictly smaller unless $U$ is indifferent between some outcomes (which may have measure zero, depending on your assumptions).

Proof: See footnote^[3]; nothing surprising.

This motivates considering only partitions in the context of optimality for a single $U$ (the remainder of section 2.)

2.2 Optimal partitions $P_i,P_j$ can be orthogonal

Fixing $U$, write $P_i$ for the optimal partition of size $i$. Then, there’s not necessarily any relationship between $P_i$ and $P_j$ for $i\ne j$.

For instance, take $X=\left[\mathrm{1..3}\right]\times \left[\mathrm{1..4}\right]$, $A=[1,2]\times \left[\mathrm{1..4}\right]$ and $$\begin{array}{cc}U(x_{i,\ast },a_{1,i})& =4\\ U(x_{\ast ,i},a_{2,i})& =5\\ \text{otherwise}U(\ast ,\ast )& =0\end{array}$$ Then the optimal partition of size 3 is $\left\{\right\{(1,j):\forall j\},\{(2,j):\forall j\},\{(3,j):\forall j\left\}\right\}$, while the optimal partition of size 4 is $\left\{\right\{(i,1):\forall i\},\{(i,2):\forall i\},\{(i,3):\forall i\},\{(i,4):\forall i\left\}\right\}$ (easy exercise: show this). These partitions are orthogonal—in other words, they have zero mutual information. This example can be extended to make any finite number of optimal $P_j$ orthogonal.

2.3 Not every $P_1,...P_n$ is (uniquely) optimal for any $U$

Then, can any sequence of partitions whatsoever be optimal for the right $U$? Not uniquely so, at least (again ruling out e.g. the trivial case where U is constant and all partitions are optimal). Consider the following counterexample:

For $P_4$ to be uniquely optimal, it must be that subdividing red provides more marginal benefit than subdividing green or blue, but optimality of $P_5$ implies the opposite.

The general principle here: if two strictly size-optimal partitions both involve partitioning the same subset of $X$ into $k$ pieces, then they must do so the same way.

2.4 Strict monotonicity of $u_i$

What about how the attainable utility varies with $\left|M\right|$? Write $u_k$ for the maximum expected utility attainable when $\left|M\right|=k$, and $u_{max}$ for the global maximum ${max}_k{u}_k$. Obviously for $i<j$, we have $u_i\le u_j$; and in fact, if $u_i<u_{max}$, then $u_i<u_j$.

An easy way to see this is to consider $N_{max}=\left|\text{Range}\right(x\to {argmax}_{A}U(x,a)\left)\right|$ - the number of distinct optimal actions. Then if $k<N_{max}$, we can always improve $u_{k+1}$ by using the additional partition to move some $k$ from a suboptimal action to an optimal action. If $k\ge N_{max}$ then we can assign every $x$ it’s optimal action and $u_k=u_{max}$.

Naively, you might imagine there could be a scenario where more information doesn’t help until you reach some threshold. For instance, suppose you can either a) do nothing or b) enter a 5-bit password to attempt to open a safe; the safe explodes if you fail, so it’s only worth attempting if you definitely know the full password. However, even though you need to know the whole password to change your policy, you can convey the whole password a smaller fraction of the time. Even with only two states for $M$, you can have one state represent ‘password unknown, do nothing’, and the second (rare) state represent ‘the password is exactly 00000, so enter that’. This result shows that to get that kind of thresholding behavior, we’d need to add something else to the model, i.e. that constrains the allowed partitions.

2.5 Marginal return can increase

We might expect that you use your first split to encode the most important information, the next split to encode the next most important information, etc., so that the marginal return to channel capacity is always decreasing. It turns out this is not necessarily the case. Consider the following counterexample:

$X=\left[\mathrm{1..2}\right]\times \left[\mathrm{1..3}\right]$, $A=\left[\mathrm{0..2}\right]\times \left[\mathrm{1..3}\right]$ and $$\begin{array}{cc}U(x_{\ast ,\ast },a_{0,\ast })& =0\\ U(x_{i,\ast },a_{1,i})& =8\\ U(x_{\ast ,i},a_{2,i})& =21\\ \text{otherwise}U(\ast ,\ast )& =-1000\end{array}$$ With a single part, the best we can do is $A_0$ everywhere, so $u_1=0$; with three parts, we can divide along the environment’s second dimension and get 21 everywhere, so the average is $u_3=21$. But we’ve chosen the utilities so the best we can do with two parts is to divide along the first dimension, achieving 8 average utility: any policy for a 2-part partition which uses a $l=2$ action must either suffer −1000 for some $x$, or divide between $a_{0,0}$ and a single $a_{2,i}$ part on at most ¹⁄₃ of $X$, achieving at most ²¹⁄₃ = 7 utility. So $u_2=8$, and $u_2-u_1=8<13=u_3-u_2$.

3. Dominance over all $U$

Now that we’ve gotten a taste of the constraints that exist for individual $U$ and $\varphi$, let’s move to our main topic: common preferences among multiple $U$. Start from the opposite extreme: what preferences over information structures are shared by all possible utility functions?

This turns out to be exactly the question answered by the foundational result about information structures: the Blackwell Informativeness Theorem.

Unsurprisingly, given the strength of the requirement and lack of structure in our setting, these convergent preferences will be sparse. The result is not exactly a no-free-lunch theorem, since some information structures are indeed better than others, but it’s not so far off in spirit: we can think of it stating that only the minimum possible preferences are shared.

Based on the barriers this case reveals, we’ll then extend our setting so that non-trivial convergence is possible.

3.1 The Blackwell order

What is required for $\phi$ to be preferred to ${\phi }^{\prime }$ for all $U$? One clear sufficient condition is that given $\phi$, we can use it to build ${\phi }^{\prime }$ by postprocessing—then we can imitate any policy for ${\phi }^{\prime }$. And we might expect that this is also necessary: if not, ${\phi }^{\prime }$ must not distinguish some situations which $\phi$ distinguishes, and we can construct some $U$ that cares about the difference.

This is what the Blackwell Informativeness Theorem confirms; and once we cast it in just the right terms, the proof falls out almost immediately.

Since some of our later results have very similar proofs, we’ll reproduce the argument here. We follow the presentation of the paper “Blackwell’s Informativeness Theorem Using Diagrams” and especially of this twitter thread.

We’ll need one additional definition:

Definition: Attainable outcomes

Define $D\left(\phi \right)=\{(X,\pi \left(\phi \right(X\left)\right)):\pi \in M_{\phi }\to \Delta A\}\subseteq \Delta (X\times A)$ to be the attainable outcome distributions of $\phi$.

That is, an element $d\in D\left(\phi \right)$ is a probability distribution over $(X,A)$ with $p_d(x,a)=p_{\pi }\left(a|m\right)p_{\phi }\left(m|x\right)p\left(x\right)$ for some $\pi$. Note that we can equivalently consider $D\left(\phi \right)$ to be the set of stochastic maps $\pi \circ \phi$ themselves, i.e. the likelihoods $p_{\pi }\left(a|m\right)p_{\phi }\left(m|x\right)$, up to isomorphism by application to our uniform prior on $X$^[4]. In other words, $D\left(\phi \right)\cong \overline{D}\left(\phi \right)=\{\pi \circ \phi :\pi \in M\to \Delta A\}\subseteq \{X\to \Delta A\}$; these can in turn be considered vectors in $(\Delta A{)}^X$.

Then we can state the theorem:

Blackwell Informativeness Theorem

The following are equivalent:

1. There exists some stochastic map $\gamma$ such that ${\phi }^{\prime }=\gamma \circ \phi$. We also say this as: ”${\phi }^{\prime }$ is a garbling of $\phi$”.

2. $D\left({\phi }^{\prime }\right)\subseteq D\left(\phi \right)$ for any $A$

3. For any $A$ and $U$, we have ${\phi }^{\prime }{\le }_U\phi$.

When these hold, write ${\phi }^{\prime }{\le }_{BW}\phi$.

Proof:

$1)⟹2)$: If you ${\phi }^{\prime }$ is a garbling of $\phi$, and you have $\phi$, you can just do the garbling yourself to get ${\phi }^{\prime }$; so you can certainly act according to ${\phi }^{\prime }$ if you want to.

Let ${\phi }^{\prime }=\gamma \circ \phi$ be a garbling of $\phi$.

By definition, for any $d^{\prime }\in \overline{D}\left({\phi }^{\prime }\right)$ there is a ${\pi }^{\prime }$ s.t. $d^{\prime }={\pi }^{\prime }\circ {\phi }^{\prime }$. Then defining $\pi ={\pi }^{\prime }\circ \gamma$, the diagram commutes^[5], showing $d^{\prime }\in \overline{D}\left(\phi \right)$.

$2)⟹1)$: If $\phi$ can achieve whatever ${\phi }^{\prime }$ can, then in particular it can achieve “be ${\phi }^{\prime }$”, which makes ${\phi }^{\prime }$ a garbling.

Take $A=M^{\prime }$ - then for ${\pi }^{\prime }={id}_{M^{\prime }}$, 2) says there is a $\pi$ such that the diagram commutes. Then ${\phi }^{\prime }=\pi \circ \phi$ is a garbling of $\phi$.

$2)⟹3)$: If $D\left(\phi \right)$ contains $D\left({\phi }^{\prime }\right)$, then it gives strictly more options, so your best option is at least as good - $\forall U:{max}_{D\left({\phi }^{\prime }\right)}U\left(d\right)\le {max}_{D\left(\phi \right)}U\left(d\right)$.

$3)⟹2)$: This is the one step which isn’t immediate from the definitions—it’s instead an application of a separation theorem.

Suppose $2)$ is false, say $d^{\prime }\in D\left({\phi }^{\prime }\right)\setminus D\left(\phi \right)$. Now $D\left(\phi \right)$ and $\left\{d^{\prime }\right\}$ are convex and compact, so the hyperplane separation theorem states that we can find a $u\in R^{\left|X\right|\cdot \left|A\right|},b\in R$ such that $u^{T}d<b$ for all $d\in D\left(\phi \right)$ and $u^T{d}^{\prime }>b$ for $d^{\prime }\in \left\{d^{\prime }\right\}$ - a hyperplane with normal vector $u$ separating them. But this is exactly a $U$ where $\phi$ does worse than ${\phi }^{\prime }$, contradicting 3). $\square$

Notice that the equivalence classes of ${\le }_{BW}$ are exactly what we said are really ‘the same’ information structure: one way to see this is to note that $\gamma A=B$ and ${\gamma }^{\prime }B=A$ imply ${\gamma }^{\prime }={\gamma }^{-1}$, and the only stochastic matrices with an inverse within the stochastic matrices are permutation matrices, which just interchange the states of $M$. So the only way one information structure can be universally preferred to a second is if one ‘contains’ the other, and two information structures which all $U$ are indifferent between are ‘the same one’.

When $\phi$ and ${\phi }^{\prime }$ are deterministic, we can easily show that this is equivalent to $P_{\phi }$ being refinement of $P_{{\phi }^{\prime }}$ as a partition (from property 1, or by explicitly constructing $U$ for property 3). For this case, we can then state the result as: the partial order of dominance of deterministic information structures is equal to their standard partial order as partitions.

We can also remark that this is a strictly stronger requirement than $\phi \left(X\right)$ having more mutual information with $X$ than ${\phi }^{\prime }\left(X\right)$. While $I(X;{\phi }^{\prime }(X\left)\right)\le I(X;\phi (X\left)\right)$ is a necessary condition for ${\phi }^{\prime }{\le }_{BW}\phi$, it is not a sufficient one - $\phi$ must contain “a superset of” ${\phi }^{\prime }$’s information about $X$, rather than simply “more” information about $X$.

Some corollaries for partitions:

• No partition of size $k<\left|X\right|$ dominates any other (non-degenerate) partition of size $\ge k$

• $P$ is always the unique smallest partition dominating itself

• The discrete partition $\left\{\right\{x\}:x\in X\}$ uniquely dominates all other partitions (and is the unique smallest information structure which dominates all others in general).

3.2 What’s missing

So we’ve seen that by default, there’s no nontrivial superiority of information structures over all possible utility functions. What’s missing to allow for more interesting convergence behavior? Two things:

1. A way to break the symmetry between different information structures. There’s nothing to break the symmetry between different $x$ or actions, and so (for example) only part-sizes to break the symmetry between different partitions. This fundamentally prevents strong convergence to exact information structures of a given size.

2. A non-extremal aggregation over utility functions. Looking at superiority over all utility functions is equivalent to considering only the worst-case. It also allows only boolean, not scalar, comparisons: $\phi$ either is or isn’t better than ${\phi }^{\prime }$ for all $U$. Instead, looking at ‘most’ (and ‘how many’) utility functions will let us make finer distinctions between the usefulness of information structures.

The symmetry is the sharpest obstacle to stronger convergence, so we’ll address 1) first.

4. Adding structure

4.1 Adding a ‘system’: the Good(er) Regulator Theorem

As a first example of what happens when we introduce more structure—and because it leads us another existing theorem—let’s ask: what if we restrict the distinctions between $x$ which the utility function is allowed to care about? What does this change about the preference dominance order?

The discrete version of this idea is for $U$ to treat some values of $X$ identically—at which point we can consider $U$ to be defined on the equivalence classes of $X$, giving the ‘causal’ structure $X\stackrel{\sim }{\to }C\to C\times A\stackrel{U}{\to }R$. What’s the probabilistic version?

Well, we can simply take a stochastic map to the intermediate object, giving the structure $X\stackrel{f}{\to }\Delta S\to \Delta (S\times A)\stackrel{U}{\to }R$, for some set $S$ and stochastic map $f$ (which we’ll assume to be surjective^[6] for convenience; otherwise reduce $S$). This is a kind of dimensionality reduction (as even if $\left|S\right|>\left|X\right|$, certainly $\dim \left(\text{Im}\right(f\left)\right)\le \dim \left(X\right)$ as a vector space).

Now our setting looks like this:

This may look familiar if you’ve read “Fixing The Good Regulator Theorem”:

This is exactly the same setting, up to some choices of representation—the extra node $Y$ just encodes the choice to look at a fixed encoder which must be shared^[7] by many^[8] policies and utility functions.

Taking the Good Regulator perspective, we can think of $S$ being the system we care about, with $f$ describing how its states depend on the observable $X$. The question the Gooder Regulator Theorem asks is in fact just: given a system $(S,f)$, what’s the minimal information structure that dominates all others?

The Gooder Regulator theorem tells us that it should in some sense be the posterior distribution $P(S\mid X)$ - we’ll revisit this in a moment. (Meanwhile, in the discrete case: notice that $X\to C$, as an equivalence relation, can also just be viewed as partition $P_C$ of $X$. We should expect that the optimal partition will just be $P_C$.)

What about the full preference dominance order, like the Blackwell order gives us for the base setting?

4.1.1. The GR-Blackwell order

We can indeed give a characterization of this order which parallels the Blackwell Informativeness Theorem.

The key idea is to look at the new equivalence classes of $\phi$ with respect to $U\left(\phi \right)$: namely, the expected utility now depends exactly on what distribution of $s\in S$ gets assigned each action $a\in A$. This suggests that we should look everything in terms of distributions over $S$.

We then find that for ${\phi }^{\prime }$ to be dominated by $\phi$, it’s not necessary for ${\phi }^{\prime }$ to be a garbling of $\phi$ (though this is of course still sufficient), but only necessary that “the mapping of system states to signals implied by ${\phi }^{\prime }$” is a garbling of “the mapping of system states to signals implied by $\phi$”. They will further be equivalent if and only if they imply the same joint distribution of $(S,M)$.

Formally,

GR setting definitions

Define $F\left(\phi \right):\Delta S\to \Delta M$ by $F\left(\phi \right)\left(s\right)=\frac{{\sum }_x\left[f{]}_{xs}\phi \right(x)}{{\sum }_x[f{]}_{xs}}$ - the (stochastic) map induced by $\phi$ specifying ‘how much do we map ($x$ that result in) state $s$ to signal $m$’.

Define the attainable $S$-outcome distributions to be:

$D_S\left(\phi \right)=\{(S,\pi \left(F\right(\phi \left)\right(S\left)\right)){]}_S:\pi \in (M\to \Delta A)\}\subseteq \Delta (S\times A)$ or isomorphically,
${\overline{D}}_S\left(\phi \right)=\{\pi \circ F(\phi ):\pi \in M\to \Delta A\}\subseteq \{S\to \Delta A\}$

Just as the elements of $D\left(\phi \right)$ correspond to stochastic maps $X\to \Delta A$ formed by $\pi \circ \phi$, the elements of $D_S\left(\phi \right)$ correspond to stochastic maps $\Delta S\to \Delta A$ formed by $\pi \circ F\left(\phi \right)$.

Then we have:

The GR-Blackwell order

The following are equivalent:

1. There exists a stochastic map $\gamma$ such that $F\left({\phi }^{\prime }\right)=\gamma \circ F\left(\phi \right)$

2. $D_S\left({\phi }^{\prime }\right)\subseteq D_S\left(\phi \right)$ for any $A$

3. For any $A$ and $U:S\times A\to R$, we have ${\phi }^{\prime }{\le }_U\phi$.

When they hold, write ${\phi }^{\prime }{\le }_S\phi$.

This is the same result as the Blackwell Informativeness Theorem, but after applying a functor $F$ which quotients out the equivalences induced by $f:X\to \Delta S$ (while leaving everything else unchanged):

And once we’ve stated it like this, if we look at the proof of the regular Blackwell order, we can see that each part carries over almost exactly. Substitute $S$ and $F\left(\phi \right)$, and you’ve proved this new equivalence.

The only place an additional step is added is in (3)=>(2). Our separation theorem gives us (towards contradiction) a ${\pi }^{\prime }$, $u\in R^{S\times A}$ such that $$\sum _{S,A}[{\pi }^{\prime }\circ F({\phi }^{\prime }){]}_{sa}{u}_{sa}>\sum _{S,A}[\pi \circ F\left(\phi \right){]}_{sa}{u}_{sa}$$ for all $\pi$.

Meanwhile, $U\left(\phi \right)$ is ${max}_{\pi }{\sum }_{X,A}[\pi \circ \phi {]}_{xa}U(f\left(x\right),a)$; but after some rearranging, this can be written ${max}_{\pi }{\sum }_{S,A}p\left(s\right)[\pi \circ F(\phi \left){]}_{sa}U\right(s,a)$, where $p\left(s\right)={\sum }_X[f{]}_{xs}$.

This extra factor of $p\left(s\right)$ is then no problem for the proof—we can just take $\overline{U}(s,a)=u_{sa}/p\left(s\right)$ (instead of $\overline{U}(s,a)=u_{sa}$) as the contradictory utility function preferring ${\phi }^{\prime }$. But it highlights how a) the likelihood of states doesn’t impact preferences over all utility functions (though it can matter for other questions), and b) even if we start by considering only a uniform prior on $X$, the introduction of the system $S$ allows us get back a weighted version of our setting. (Indeed, if we redefine ‘information structures’ to be equivalence classes of $F\left(\phi \right)$, this is exactly what we would have—but with some restrictions on the set of possible information structures if $f$ is nondeterministic).

Alternate precomposition form

In fact, another reflection of the same scaling invariance: while we normalized $F\left(\phi \right)$ to be a proper stochastic map (i.e. column-stochastic matrix) by dividing by ${\sum }_x[f{]}_{xs}$, the equation $F\left({\phi }^{\prime }\right)=\gamma \circ F\left(\phi \right)$ will hold iff the same equation holds^[9] with unnormalized $F\left(\phi \right)$. Inspecting our equation for $F\left(\phi \right)\left(s\right)$, we can observe that this unnormalized form of $F\left(\phi \right)$ as a matrix is actually just $\phi f^T$. This gives us a nice alternate statement of prong 1), equivalent to all the rest:

1′) There exists a stochastic matrix $\gamma$ such that ${\phi }^{\prime }{f}^T=\gamma \phi f^T$.

4.1.2 Gooder Regulator revisited

Let’s explicitly show how the Gooder Regulator Theorem relates to the GR-Blackwell order, and the minimal information structure which weakly dominates all others.

Let $X_S=\left\{f\right(x):x\in X\}$ - as a set, so potentially $\left|X_S\right|<\left|X\right|$. And to make it clearer what role $M$ plays: we’ll take $M=\{1,...,\left|X_S\right|\}$ to be an enumeration of it, rather than $X_S$ itself; write $M\left(x\right)$ for the index of $f\left(x\right)$ in $M$.

Define ${\phi }_S$ by ${\phi }_S\left(x\right)=M\left(x\right)$ (deterministically).

Then the Gooder Regulator Theorem is to show that for all information structures $\phi$:

1. $\phi {\le }_S{\phi }_S$.

2. If $\left|\phi \right|<\left|{\phi }_S\right|$, then $\phi {<}_S{\phi }_S$ (i.e. additionally $\phi {\ngeqq }_S{\phi }_S$).

We’ll give a quick proof in these terms—not too different from Wentworth’s, but taking advantage of the fact that ${\phi }_S$ allows for globally optimal actions.

Proof:

1) For the first part, we can just remark that for any $U$, the globally optimal action for $x$ is ${argmax}_{A}U\left(f\right(x),a)$^[10], so the globally optimal behavior $A\left(x\right)={argmax}_{A}U\left(f\right(x),a)$ is achievable with a policy from ${\phi }_S$. Hence ${\phi }_S$ dominates any possible information structure.

2) For the second part: if $\left|\phi \right|<\left|{\phi }_S\right|$, by pigeonhole there necessarily exists an $m\in M_{\phi }$ such that $[\phi {]}_{x_{1}m},[\phi {]}_{x_{2}m}>0$ with $f\left(x_1\right)\ne f\left(x_2\right)$.

It suffices to find a pair $A,U$ such that the optimal action for $x_1$ is different from the optimal action for $x_2$ - then $U\left(\phi \right)$ will be lower than the global optimum, so $\phi {<}_U{\phi }_S$ and $\phi {\ngeqq }_S{\phi }_S$.

Hence we need vectors $v_1=\left(U\right(s_i,a_1){)}_i$, $v_2=\left(U\right(s_i,a_2){)}_i$ such that $$f\left(x_1\right)\cdot v_1>f\left(x_1\right)\cdot v_{2}f\left(x_2\right)\cdot v_1<f\left(x_2\right)\cdot v_2$$

Taking $v_1=f\left(x_1\right)/||f\left(x_1\right)||$, $v_2=f\left(x_2\right)/||f\left(x_2\right)||$, we have the desired result. $\square$

Where’s the posterior?

What here is isomorphic to “the posterior distribution on $S$ given $X$”, $(s\mapsto P[S=s\mid X])$, promised by the Gooder Regulator theorem?

It’s certainly not the states of $M$, in isolation - $M$ is just an arbitrary set with the right cardinality. It’s more reasonable to say that it’s ${\phi }_S$ (a.k.a. $M\left(X\right)$). Indeed, ${\phi }_S:X\to M$ and $x\to (s\mapsto P[S=s\mid x])$ do induce the same equivalence classes of $x$, so if we look at the posterior as a function $X\to \{P[S=s\mid x]\}$ (n.b. not as $X\to \Delta S$), then they are isomorphic as $\text{Set}$ functions.

However, this isn’t necessarily the equivalence we’re looking for. Different systems $(S,f)$ with different posteriors can induce the same optimal ${\phi }_S$ - as again, they just need to have the same equivalence classes of $x$. That is, there’s no isomorphism between $S\mapsto (s\mapsto P[S=s\mid x])$ and $S\mapsto {\phi }_S$. Nor are systems yielding the same optimal ${\phi }_S$ necessarily ‘effectively identical’ - they can induce different overall preorders over information structures, with e.g. different (and even different numbers of) $k$-optimal information structures. Contra Wentworth, the “model $M\left(x\right)$” (a.k.a ${\phi }_S$) by itself does not contain all of the information from the posterior.

That said, when we consider a single agent that can optimally satisfy any $U:S\times A\to R$, it is still reasonable to say the agent as a whole must model the Bayesian posterior. This is because it must ‘know’ what distribution of states each signal $m$ corresponds to in order to choose the optimal policies for each $U$ (we can easily show this: the optimal policy for some $U$ would shift if $p(s\mid m)$ were different). The distinction is just that this information doesn’t have to be propagated from $X\to M$; it doesn’t have the type “something which must be a function of the observation”. We can think of it as (allowed to be) precomputed, or built into the agent. Regardless, really it is the pair $({\phi }_S,U\mapsto {\pi }_U)$ which must contain the full information from the posterior.

This is a bit of a subtle distinction, but the kind which can be important.

4.1.3 Convergence is still limited

Has this additional structure allowed for the possibility of natural abstractions, given the right system $(S,f)$? Generally, does this setting allow us to model the kind of preference structures over abstractions that we’re interested in? Not entirely.

The reduction by $F$ does introduce additional convergence—and asymmetric convergence, as in general $F$ will differentially expand the equivalence classes and dominated sets of different information structures. But, especially in the ‘universal preference’ setting, we’re still restricted by the identical role each $s\in S$ must play in the outcomes.

One reflection of this is that for any $k<|{\varphi }_S|$ short of the critical size allowing full information of the system, it is still impossible to make all utility functions agree on a single $\varphi$ of size $k$ which they prefer to all others of that size. Surely we ought to at least be able to represent universal preferences for a particular compression.

While a full discussion of the number of Pareto optimal information structures at a given size is out of scope, we’ll sketch a few quick properties:

• Just as in the Blackwell setting every partition of $X$ is the unique smallest partition dominating itself, when $f:X\to S$ and $\phi$ are deterministic, every $\phi$ corresponding to a partition of $S$ is the unique smallest partition dominating itself.

• In general, the number (or volume) of $k$-Pareto-optimal information structures scales at least^[11] as $O\left((\frac{|X_S|}{k})\right)$, initially growing ~exponentially as $k$ diverges from $1$ or $n$.

This limits the maximum possible convergence of abstractions. To go further, we must allow for more substantial shared structure between the utility functions.

4.2. Adding a common world

Specifically, the additional structure we need is a common world, so that different utility functions share the same relationships between context states $x$ and actions.

Define:
$W$: a set of outcomes
$T:(X,A)\to W$, a transition function
and redefine $U:W\to R$.

We’ll call the pair $(W,T)$ a ‘world’^[12].

Our setting now looks like this:

Notice that if $T$ happens to factor as $T(x,a)=\left(f\right(x),a)$ for some $S$ and $f:X\to \Delta S$, then we’re back in the Gooder Regulator setting. Generally, for this additional structure to do anything new, $T$ must be many-to-one, and the images of different actions must overlap.

4.2.1 Universal preferences in a common world

In this setting, what can we say about universal preferences?

The equivalence of prongs (2) and (3) from the Blackwell Informativeness Theorem carries over just fine.

For (2), our attainable outcome distributions are now distributions over $W$ rather than over $(X,A)$ or $(S,A)$ - all along these have been ‘distributions over whatever the utility function is allowed to care about’, and now that doesn’t (necessarily) factor in such a way.$T$ is allowed to map arbitrary combinations of $x,a$ to the same outcome.

In other words, there are still universal preferences for exactly those information structures which yield strictly larger sets of attainable outcomes, but with outcomes now coming through the world.

However, prong (1) is trickier. While it’s easy to find sufficient conditions for $\phi$ to yield more attainable outcomes than ${\phi }^{\prime }$ (as the conditions from the original or good-regulation orders still work), it’s harder to give meaningful necessary conditions. This is essentially because $x$ and $x^{\prime }$ can yield some of the same outcomes, and be equivalent for some purposes, without being globally equivalent. We can say that for all ${\pi }^{\prime }$, there must exist some $\pi$ such that $T\circ ⟨{id}_X,\pi \circ \phi ⟩=T\circ ⟨{id}_X,{\pi }^{\prime }\circ {\phi }^{\prime }⟩$ but not necessarily that $\phi$ and ${\phi }^{\prime }$ have some simpler linear relationship. While there is likely more to say here, I don’t know that (1) has a straightforward equivalent.

Then for now all we have is:

Proposition:

Define $D_W\left(\phi \right)=\left\{{\sum }_X{\sum }_A\right[T{]}_{\left(xa\right)w}[\pi \circ \phi {]}_{xa}:\pi \in (M\to \Delta A\left)\right\}\subseteq \Delta W$

The following are equivalent:

• $D_W\left({\phi }^{\prime }\right)\subseteq D_W\left(\phi \right)$ for any $A$

• For any $A$ and $U$, we have ${\phi }^{\prime }{\le }_U\phi$

When they hold, write ${\phi }^{\prime }{\le }_W\phi$.

The fact that universal preferences are less simple for arbitrary $W,T$ is exactly why they can now start to exhibit more different behaviors.

5. Example (universal) convergence behaviors

On that note: let’s now look at some possible convergence behaviors, as we vary the world. It’s possible to control these behaviors quite a bit, even if we still only consider the preferences shared by all utility functions.

5.0 Redux: simple divergence and simple convergence

First, the convergence behaviors of the Blackwell and good-regulator settings are of course still possible. This will happen if our ‘extra shared structure’ doesn’t actually add structure after all, or if it adds only GR-type structure.

That is, if $W=X\times A$ with $T(x,a)=(x,a)$, then we are in the base setting and there is no “non-trivial” convergence. Similarly, if there exist $S$, $f:X\to \Delta S$ such that $W=S\times A$ and $T(x,a)=\left(f\right(x),a)$, then we are in the GR setting with “simple” convergence at the critical size $\left|X_S\right|$, and variable but rapid divergence below.

5.1 Multi-capacity convergence

A new behavior in this setting is universal convergence at multiple scales.

Consider

$$\begin{array}{cc}X& =\left[\mathrm{1..4}\right]\times \left[\mathrm{1..5}\right]\\ A& =\left[\mathrm{1..2}\right]\times \left[\mathrm{1..5}\right]\times \left[\mathrm{1..10}\right]\\ W& =\left\{\varnothing \right\}\cup \left[\mathrm{0..10}\right]\end{array}$$ with $$\begin{array}{cc}T(x_{i,\ast },a_{1,i,r})& =0.99\ast w_r+0.01\ast w_{\varnothing }\\ T(x_{\ast ,j},a_{2,j,r})& =w_r\\ \text{otherwise}T(\ast ,\ast )& =w_{\varnothing }\end{array}$$

The environment has two dimensions, one of size 4 and one of size 5. Each action $a_{k,c,r}$ has a ‘kind’ $k$, a ‘condition’ $c$, and a ‘target’ $r$ - when value of $c$ matches the $k$th dimension of the state, the result is the ‘target’ state $w_r$, otherwise the result is the null state $w_{\varnothing }$. However, actions of kind 1 are fallible, and have a 1% chance to map to null state anyways. That is: the first dimension is smaller, so its conditions can be matched using a smaller partition, but it is less reliable.

In this setting, dividing along the first dimension is uniquely dominant among partitions of size 4, while dividing along the orthogonal(!) second dimension is uniquely dominant among partitions of size 5. For any state you might prefer, those partitions allow you to get the most of it (and there is no tradeoff among less-preferred states). This example can be extended to get arbitrarily long orthogonal chains of uniquely dominant partitions at different sizes.

We can also imitate this behavior using a fully deterministic $T$^[13].

5.2 Capacity-dependent convergence and divergence

We can also have different convergence behaviors at different scales—including abstractions that converge at one capacity, but diverge non-trivially at larger sizes.

Consider

$$\begin{array}{cc}X& =\left[\mathrm{1..4}\right]\times \left[\mathrm{1..10}\right]\\ A& =\left[\mathrm{1..2}\right]\times \left[\mathrm{1..10}\right]\\ W& =\left\{\varnothing \right\}\cup \left[\mathrm{0..10}\right]\end{array}$$ with $$\begin{array}{cc}T(x_{i,\ast },a_{1,i,r})& =0.99\ast w_r+0.01\ast w_{\varnothing }\\ T(x_{i,r},a_{2,i,r})& =w_r\\ \text{otherwise}T(\ast ,\ast )& =w_{\varnothing }\end{array}$$

Again dividing along the first dimension is uniquely dominant among partitions of size 4. However, there is no partition of size 5 (or 6..13) which dominates all others: if $U$ most prefers $w_r$, then it will most prefer separating $x=(i,r)$ into it’s own part. By size 14, the information structure can fully discriminate the states of the environment with different response characteristics, and there is once again a single dominant information structure.

We can glue multiple copies of this and the previous example together—leading to convergence and divergence at different scales of our choice.

5.3 Discussion

Overall, by funnelling all $U$ through the same outcomes $W$, many of the behaviors that could hold for a single utility function in 2. can now be made to hold for all utility functions in an appropriate world.

A few things to highlight about example 5.2:

• It leads to convergence at an agent capacity much smaller than perfect information about the system. (Suggestively, the setting ‘almost’ factors as a GR-type system at that smaller size, plus some additional restrictions on the allowed $U$)

• With the values we’ve chosen, the returns to capacity past that point are comparatively flat, and account for only a small share of the total attainable utility (for any $U$ on this environment).

• The optimal partitions of size 5+ are all refinements of our convergent partition at size 4.

This is the kind of behavior we’d need to see for practically useful natural abstractions, and given the right (bounds on a) cost function for capacity would lead to the same or similar information structures for many $U$.

Of course, this just shows ‘if we deliberately force our environment to exhibit convergent abstractions, then we get convergent abstractions’. We can also have no convergence, or convergence within a particular capacity range that shifts to a different convergent abstraction at a higher capacity.

This is all right and proper—the existence of natural abstractions is supposed to depend on specific kinds of regularity in the environment. This version of the formalism has made it possible to get convergence, but the real question is what realistic conditions on the environment would produce what behaviors. We’ve also confirmed that it’s at least possible to come up with environments which produce abstractions that depend on the agent’s capacity. To claim that differently-capable agents will use the same abstractions (if we care to^[14]), we’d need to find conditions which rule that back out.

The shape of the examples also hints that while it’s possible for abstractions of different sizes to be unrelated, it’s probably not too hard to come up with conditions such that they ‘usually’ aren’t. At the same time, it’s not clear that realistic conditions would imply this never happens.

6. Beyond universal preferences

But before we go further on that front, let’s develop a more granular and more practical means of comparison than ‘preferred by all utility functions’. We want to be able to notice when ‘most’ utility functions share a preference, even if not strictly all.

If we want to talk about preferences ‘many’ or ‘most’ utility functions, we need a way to measure them. What should this be? Well, for any given application, what we really care about is the probability measure coming from our beliefs—for example, if we’re interested in a potential agent created by some training procedure, then we care about our subjective distribution on $U$ given that training procedure. But we don’t know at this point what that will be, so we are interested in results that we can either parameterize by our probability distribution, or that hold for some class of distributions.

It’s… not actually quite redundant to look at what’s true for all distributions, depending on what questions we ask; but for the most part this is covered by our discussion of what’s true for all utility functions (consider the trivial distributions which assign all measure to a given $U$).

Our first step will instead be to look at ‘uninformative’ distributions which are invariant under particular symmetries of the problem. To do so, we’ll apply ideas from Turner’s power-seeking work.

We won’t exactly follow the formalism of Parametrically Retargetable Decision-Makers Tend To Seek Power, for the key reason that their central relation ${\ge }_{\text{most:}\Theta }^n$ describing “inequalities which hold for most orbit elements” is not transitive. But several parts of this section will closely parallel their results (and at one point we’ll directly apply a lemma from this paper). Apart from the issue of transitivity, we’ll make some slightly different choices in coverage and presentation, generalizing in some ways, while (for expedience) losing generality in others. Properly reconciling these is a topic for another post.

(Side note: it would also be natural to look at the ‘maximum entropy distribution over all utility functions’ in particular. We need to be slightly careful about what exactly this is^[15], but presumably one can show that for this case, information structures are preferred in direct proportion to their mutual information with the outcomes. We won’t pursue this here, as it would take us a bit off-track, but leave it as a conjecture.)

6.1 Dominance for all distributions from a class

Once we’re measuring $U$ according to some distribution $D$, there are a few natural questions we might ask in order to compare information structures $\phi$ and ${\phi }^{\prime }$ in terms of this measure, including:

1. For what measure of $U$ is $U\left(\phi \right)\ge U\left({\phi }^{\prime }\right)$?

2. For what measure of $U$ are $\phi$ v.s.${\phi }^{\prime }$ optimal for $U$, out of some class $\Phi$?

3. How does the expected value of $\phi$ compare to ${\phi }^{\prime }$, averaging across $D$?

In particular, we might wonder: can we get preorders out of these conditions which parallel the Blackwell and Blackwell-like orders, but allow for more granular comparisons?

For 1), it turns out the answer is no: comparing $\phi$ and ${\phi }^{\prime }$ in terms of the measure for which $U\left(\phi \right)>U\left({\phi }^{\prime }\right)$ leads to a non-transitive relation even for individual (symmetric) distributions over $U$. That is, writing $\phi {>}^{!}\phi$ for $P_D\left(U\right(\phi )>U({\phi }^{\prime }\left)\right)>P_D\left(U\right(\phi )<U({\phi }^{\prime }\left)\right)$, there can be cycles^[16] with ${\phi }_1{>}^{!}{\phi }_2{>}^{!}{\phi }_3{>}^{!}{\phi }_1$, but ${\phi }_1{\nless }^{!}{\phi }_2$. The same holds for ${>}^{!}$ defined as $P_D\left(U\right(\phi )>U({\phi }^{\prime }\left)\right)>c\in (0,1)$).

However, the latter two don’t have this problem: the probability $P_D\left({\mathrm{Opt}}_{\Phi }\right(\phi \left)\right)$ that $\phi$ is optimal out of $\Phi$, and the expected utility $E_{U\sim D}[U\left(\phi \right)]$, are functions from individual information structures to $R$. Thus, for any given $D$, they induce a total preorder on them. This allows us to safely define a preorder by intersection—for example, $\phi {\ge }_C{\phi }^{\prime }$ if $E_D[U\left(\phi \right)]\ge E_D[U\left({\phi }^{\prime }\right)]$ for all $D$ in some class $C$.

We’ll then see how these preorders correspond to bounds on ratios of probability-of-optimality and expected-utility between $\phi$ and ${\phi }^{\prime }$.

6.1.1 Dominance for $G$-invariant distributions

In particular, we’ll look at the relationships between $\phi$ and ${\phi }^{\prime }$ in terms of the symmetries satisfied by $D$; that is, we take the class $C$ to be “all distributions satisfying a given symmetry”.

We can then give simple sufficient conditions and (weaker) necessary conditions for $\phi$ to be preferred to $\phi$ over all such distributions, in the sense of higher expected value and higher probability of optimality. First, $\phi$ will be always preferred to ${\phi }^{\prime }$ if we can imitate its outcomes up to a symmetry—this direction is the same essential idea as Turner’s power-seeking results. And in the opposite direction, if $\phi$ is preferred for all symmetric $D$, then each outcome attainable by all permutations of ${\phi }^{\prime }$ must be imitable by some mixture of permutations of $\phi$.

To state this formally, we’ll need a few preliminaries.

Given a subgroup $G\le S_{\left|W\right|}$ of the symmetric group with action on $R^{\left|W\right|}$, we can take it to act on utility functions, by permuting the utilities assigned to each state. This has^[17] the effect that $g\cdot U=U\circ g^{-1}$.

Then, we can define:

Definition: $G$-invariant distribution

Define a $G$-invariant distribution to be a probability distribution $D\in \Delta \left(\right\{W\to R\left\}\right)$ such that it’s density ${\mu }_D$ (a.k.a. $p_D$) satisfies ${\mu }_D(g\cdot U)={\mu }_D\left(U\right)$ for all $U\in \{W\to R\}$ and all $g\in G$.

Now, distributions $d\in \Delta W$ are just vectors in $R^W$ which sum to $1$, so we can consider the action of $G$ on them as well^[18]. We can naturally extend this action to sets of distributions over $W$, such as our attainable outcome distributions, by $g\cdot D=\{g\cdot d:d\in D\}$.

Finally, we’ll write ${\mathrm{IsOpt}}_{\Phi }(\phi ,U)=1\left(U\right(\phi )\ge {max}_{x\in \Phi }U(x\left)\right)$, so that $P_D\left({\mathrm{Opt}}_{\Phi }\right(\phi \left)\right)=E_{U\sim D}[{\mathrm{IsOpt}}_{\Phi }(\phi ,U)]$.

Then our sufficient condition (for the general $W$ setting) will be the following:

Theorem ($g$-covering of outcomes implies score-domination under $G$-invariant distributions)

Suppose $g\cdot D\left({\phi }^{\prime }\right)\subseteq D\left(\phi \right)$ for some $g$. Then:

1. ${\sum }_{g\in G}(g\cdot U)\left({\phi }^{\prime }\right)\le {\sum }_{g\in G}(g\cdot U)\left(\phi \right)$ for all $U$

2. $E_{U\sim D}[U\left({\phi }^{\prime }\right)]\le E_{U\sim D}[U\left(\phi \right)]$ for all $G$-invariant $D$

Write ${\phi }^{\prime }{\le }_{EU}^G\phi$ when these hold.
Additionally, let $\Phi$ be a set of information structures with outcomes closed under the action of $G$: if ${\phi }_a\in \Phi$, then for all $g$, $gD\left({\phi }_a\right)=D\left({\phi }_b\right)$ for some ${\phi }_b\in \Phi$.
Then:

1. ${\sum }_{g\in G}{\mathrm{IsOpt}}_{\Phi }({\phi }^{\prime },gU)\le {\sum }_{g\in G}{\mathrm{IsOpt}}_{\Phi }(\phi ,gU)$ for all $U$

2. $P_D\left({\mathrm{Opt}}_{\Phi }\right({\phi }^{\prime }\left)\right)\le P_D\left({\mathrm{Opt}}_{\Phi }\right(\phi \left)\right)$ for all $G$-invariant $D$

Write ${\phi }^{\prime }{\le }_{{\mathrm{Opt}}_{\Phi }}^G\phi$ when these hold.

In fact, this holds for the expectation of any ‘scoring function’ $\nu$ mapping a set of attainable outcomes (or information structure) and a utility function to a real number, which is

• monotonic in the set (i.e.$A\subseteq B$ implies $\nu (A,U)\le \nu (B,U)$), and

• invariant under joint action of $G$ (i.e. $\nu (g\cdot A,g\cdot U)=\nu (A,U)$).

The proof follows more or less immediately from these properties, so we leave it to an appendix.

In the other direction, our necessary condition (which we’ll state just in terms of expected utility) is:

Suppose $E_{U\sim D}[U\left({\phi }^{\prime }\right)]\le E_{U\sim D}[U\left(\phi \right)]$ for all $G$-invariant $D$.
Then ${\bigcap }_{G}D(g\cdot {\phi }^{\prime })\subseteq \mathrm{Conv}\left({\bigcup }_{G}D\right(g\cdot \phi \left)\right)$

Proof: Suppose the contrary, and take $x\in {\bigcap }_{G}D(g\cdot \phi )\setminus \mathrm{Conv}\left({\bigcup }_{G}D\right(g\cdot {\phi }^{\prime }\left)\right)$. As in the Blackwell order proof, we can take a hyperplane which separates $\left\{x\right\}$ and $\mathrm{Conv}\left({\bigcup }_{G}D\right(g\cdot {\phi }^{\prime }\left)\right)$, giving a $U$ which prefers $x$ to any point attainable by any $g\cdot {\phi }^{\prime }$. Now $x$ is attainable by each $g\cdot \phi$, so $U(g\cdot \phi )>U(g^{\prime }\cdot {\phi }^{\prime })$ for any $g,g^{\prime }$, and hence certainly ${\sum }_{G}U(g\cdot \phi )>{\sum }_{G}U(g\cdot {\phi }^{\prime })$. Therefore $E_{U\sim D}[U\left({\phi }^{\prime }\right)]>E_{U\sim D}[U\left(\phi \right)]$ for $D$ uniform over the orbit of $U$ - a contradiction. $\square$

In a narrow sense, this is strict, as we can find counterexamples with $E_{U\sim D}[U\left({\phi }^{\prime }\right)]>E_{U\sim D}[U\left(\phi \right)]$ even if $\left|G\right|-1$ of the orbits of each point in $D\left({\phi }^{\prime }\right)$ are contained by every orbit of $D\left(\phi \right)$. However, there should be a related necessary-and-sufficient characterization, giving a proper generalization of the Blackwell Informativeness Theorem; we set this aside for now.

Additional results for the Blackwell setting

To further show how this is all still connected to the Blackwell order, let’s also give conditions corresponding to prong 1) of the Blackwell Informativeness Theorem, for the traditional Blackwell setting where $W=(X,A)$ and $T=id$.

Suppose $G$ acts independently on each of $X$ and $A$ - and permutations of $A$ are in fact always irrelevant (when we allow all policies), so equivalently it acts only on $X$.

Then our sufficient condition

$g\cdot D\left({\phi }^{\prime }\right)\subseteq D\left(\phi \right)$

is equivalent to^[19]

${\phi }^{\prime }=\gamma \circ \phi \circ g$ for some stochastic map $\gamma$ and $g\in G$

This is because here the action of $G$ on an outcome distribution is just a relabelling of $X$, and we can^[20] equally perform this relabelling before applying $\phi$. In fact, this characterization will work whenever we can define an action of $G$ on $X$ and $A$ such that $G$ ‘commutes’ with $T$, in the sense that $g\cdot T(x,a)=T(g\cdot x,g\cdot a)$ (which will not in general be possible).

When $G$ is the full symmetric group $S_{\left|X\right|}$, this means that all information structures with the ‘same weight distributions’ across signals are equivalent—for deterministic partitions, this is ‘the same set of part sizes’.

Beyond the independent case, when $G$ acts arbitrarily on $X\times A$ and can permute action-values across states, then the condition on $\phi$ is less obvious, though we can at least show it is a) nontrivial, b) different from the independent case, c) likely some form of majorization condition^[21]. We also set this aside for now.

Finally, we’ll note that our necessary condition can be translated to a similar form, but doesn’t appear particularly enlightening^[22].

6.2 Quantitative dominance

At this point, we’ve found a kind of superiority weaker (and so more general) than ‘preferred for all possible utility functions’, but we still haven’t moved beyond a binary ‘preferred’ or ‘not preferred’ relationship for $\phi ::{\phi }^{\prime }$.

To do so, a natural idea is to move from simple inequalities $E_{U\sim D}[\nu ({\phi }^{\prime },U)]\le E_{U\sim D}[\nu (\phi ,U)]$ to inequalities parameterized by a scalar $c\in R$ - for example, defining ${\phi }^{\prime }{\le }_{EU}^{G,c}\phi$ by $E_{U\sim D}[U\left({\phi }^{\prime }\right)]\le c\cdot E_{U\sim D}[U\left(\phi \right)]$ and similarly for $P_D\left({\mathrm{Opt}}_{\Phi }\right({\phi }^{\prime }\left)\right)\le c\cdot P_D\left({\mathrm{Opt}}_{\Phi }\right(\phi \left)\right)$. This clearly gives another preorder, and is a perfectly reasonable way to compare information structures under a particular distribution $D$. But taking a lower bound $c$ such that this holds for all symmetric distributions $D$ is not actually interesting: if we take a trivial $D=\left\{U_1\right\}$, $U_1\left(w\right)=1$ for all $w$, then the expectations will just be equal, making $c$ always 1.

However, we can define a related relationship with less trivial bounds:

Define ${\phi }^{\prime }{\le }_{\ne \nu }^{D,c}\phi$ to mean $E_{U\sim D}[\nu ({\phi }^{\prime },U)\mid \nu (\phi ,U)\ne \nu ({\phi }^{\prime },U)]\le c\cdot E_{U\sim D}[\nu (\phi ,U)\mid \nu (\phi ,U)\ne \nu ({\phi }^{\prime },U)]$.

Note that this is equivalent to $E_{U\sim D}[\nu ({\phi }^{\prime },U)]\le c\cdot E_{U\sim D}[\nu (\phi ,U)]$ for exactly $c=1$, but not otherwise.

It’s less obvious that this relationship is transitive, since we take the expectation over a different conditional distribution for each pair $\phi ,{\phi }^{\prime }$, but it turns out to in fact be the case^[23].

For this relationship, we can define ${\phi }^{\prime }{\le }_{\ne \nu }^{G,c}\phi$ to mean ${\phi }^{\prime }{\le }_{\ne \nu }^{D,c}\phi$ for all $G$-invariant $D$, and have this be nontrivial for $c>1$.

Multiple $G$-injections

It’s also this relationship ${\le }_{\ne \nu }^{G,c}$ which we can infer by generalizing the previous result about $g$-injections (paralleling the move from simple- to multiple- retargetability in the parametric retargetability context). Intuitively, if there is not just one copy of $D\left({\phi }^{\prime }\right)$ within $D\left(\phi \right)$ via $g$, but $k$ ‘distinct’ copies under different $g_1,...,g_k$, we should be able to say that $\phi$ is not just ‘at least as good as ${\phi }^{\prime }$’ but ′$k$ times better than ${\phi }^{\prime }$′ in some way. However, we need to be a bit careful with the details.

For this part, we’ll apply a lemma from Turner & Tadepalli on “orbit-level tendencies” directly, and give the results only for ${\mathrm{IsOpt}}_{\Phi }$.

Specifically, for our family $g_1,...,g_k$ with $g_i\cdot D\left({\phi }^{\prime }\right)\subseteq D\left(\phi \right)$, we’ll additionally require their conditions^[24] for “containment of set copies” (Definition A.7):

We say that $B$ contains $k$ copies of $A$ when there exist $g_1,...,g_k$ such that
a) $g_i\cdot A\subseteq B$
b) $g_i$ are involutions: $g_i=g_i^{-1}$
c) $g_i$ fix the image of other $g_j$: $g_i{g}_j\cdot A=g_j\cdot A$ for $i\ne j$

Then by their Lemma B.9, we conclude that $$k\cdot \left|\right\{g:{\mathrm{IsOpt}}_{\Phi }({\phi }^{\prime },gU)>{\mathrm{IsOpt}}_{\Phi }(\phi ,gU)\left\}\right|\le \left|\right\{g:{\mathrm{IsOpt}}_{\Phi }({\phi }^{\prime },gU)<{\mathrm{IsOpt}}_{\Phi }(\phi ,gU)\left\}\right|$$ for any $U$ (or in their notation, ${\mathrm{IsOpt}}_{\Phi }\left(D\right({\phi }^{\prime })\mid \theta ){\le }_{\text{most:}\Theta }^n{\mathrm{IsOpt}}_{\Phi }\left(D\right(\phi )\mid \theta )$, interpreting once again $U$ as $\theta \in \Theta =R^{\left|W\right|}$). Since the value of ${\mathrm{IsOpt}}_{\Phi }$ is in $\{0,1\}$, this immediately gives us^[25] that $$k\sum _{g\in G_{\ne }}{\mathrm{IsOpt}}_{\Phi }({\phi }^{\prime },gU)\le \sum _{g\in G_{\ne }}{\mathrm{IsOpt}}_{\Phi }(\phi ,gU)$$ where $G_{\ne }=\{g\in G:{\mathrm{IsOpt}}_{\Phi }({\phi }^{\prime },gU)\ne {\mathrm{IsOpt}}_{\Phi }(\phi ,gU\left)\right\}$. Since this holds for the orbit of any $U$, we can conclude that ${\phi }^{\prime }{\le }_{\ne {\mathrm{IsOpt}}_{\Phi }}^{G,k}\phi$.

That is, we have the following theorem:

Theorem

Suppose $D\left(\phi \right)$ contains $k$ copies of $D\left({\phi }^{\prime }\right)$ via $g_1,...,g_k\in G$.
Then ${\phi }^{\prime }{\le }_{\ne {\mathrm{IsOpt}}_{\Phi }}^{G,k}\phi$.

The conclusion in english: according to any $G$-symmetric probability distribution, it is at least $k$ times more likely that $\phi$ is optimal while ${\phi }^{\prime }$ isn’t, than that ${\phi }^{\prime }$ is optimal while $\phi$ isn’t.

While somewhat limited, this result gives us an initial scenario where we can establish a kind of quantitative dominance of one information structure over another, for a broad range of distributions of utility functions. The theorem can surely be improved; but for meaningfully stronger conclusions, we’d require stronger assumptions about the environment or the utility function distributions.

6.3 Blackwell-like relations correspond to ratio bounds

So far, we’ve framed our results in terms of various preorders on information structures. Mostly the intuitive meanings of these “$\le$” relationships are basically their definitions. But to make it clearer what they mean in terms of “convergence” of abstractions, especially for the quantitative relations, we’ll mention another interpretation here: each relationship ${\phi }^{\prime }{\le }_{\ast }\phi$ corresponds to a lower bound on a ratio of some quantities related to $\phi$ and ${\phi }^{\prime }$.

The original Blackwell relation ${\phi }^{\prime }{\le }_{BW}\phi$, which states that $U\left({\phi }^{\prime }\right)\le U\left(\phi \right)$ for all $U$, implies that $\frac{U\left(\phi \right)}{U\left({\phi }^{\prime }\right)}\ge 1$ for all $U$ - with the corollary that e.g.$\frac{P_D\left({\mathrm{Opt}}_{\Phi }\right(\phi \left)\right)}{P_D\left({\mathrm{Opt}}_{\Phi }\right({\phi }^{\prime }\left)\right)}\ge 1$ for any $D$. Likewise, ${\le }_S$ and ${\le }_W$ imply the same for $U$ (and distributions of $U$) defined on a system $S$ or a world $W$.

Then, a relation ${\phi }^{\prime }{\le }_{\nu }^G\phi$ corresponds to the bound $\frac{E_D[\nu \left(\phi \right)]}{E_D[\nu \left({\phi }^{\prime }\right)]}\ge 1$ for all $G$-invariant $D$.

And moving to our quantitative relations corresponds to changing the bound from $1$ to $c$. The hypothetical ${\phi }^{\prime }{\le }_{\nu }^{c,G}\phi$ would correspond to a bound $\frac{E_D[\nu \left(\phi \right)]}{E_D[\nu \left({\phi }^{\prime }\right)]}\ge c$ - for $\nu$ the probability of an event (such as optimality with respect to $\Phi$), this corresponds to a lower bound on the likelihood ratio of the events, e.g. $\frac{P_D\left[\mathrm{Opt}\right(\phi \left)\right]}{P_D\left[\mathrm{Opt}\right({\phi }^{\prime }\left)\right]}\ge c$. However, these are exactly the relationships that we found it wasn’t possible to guarantee (for $c>1$) for arbitrary symmetric distributions; which means there is no nontrivial bound on likelihood ratios with our assumptions (recalling e.g. the trivial but symmetric $D=\left\{U_1\right\}$, $U_1\left(w\right)=1$). It would be possible to get such bounds, but it would require a different type of assumption on $D$.

Finally, our substitute relations like ${\phi }^{\prime }{\le }_{\ne {\mathrm{IsOpt}}_{\Phi }}^{G,c}\phi$ correspond to bounds on more unusual ratios of the form: $\frac{E_{U\sim D}[\nu (\phi ,U)\mid \nu (\phi ,U)\ne \nu ({\phi }^{\prime },U)]}{E_{U\sim D}[\nu ({\phi }^{\prime },U)\mid \nu (\phi ,U)\ne \nu ({\phi }^{\prime },U)]}\ge c$ When $\nu$ is a binary indicator of an event like ${\mathrm{IsOpt}}_{\Phi }$, this gives likelihood ratios like: $\frac{P_D\left({\mathrm{Opt}}_{\Phi }\right(\phi )\wedge \neg {\mathrm{Opt}}_{\Phi }({\phi }^{\prime }\left)\right)}{P_D\left({\mathrm{Opt}}_{\Phi }\right({\phi }^{\prime })\wedge \neg {\mathrm{Opt}}_{\Phi }(\phi \left)\right)}\ge c$.

This unusual likelihood ratio highlights that “contains multiple copies”-style relations don’t necessarily give us as strong of a form of convergence as we might hope: while they can tell us $A\text{and not}B$ is at least $k$ times more likely than $B\text{and not}A$, they can’t unconditionally tell us that e.g.$p_{\text{selection}}\left(B\right)$ is $k$ times more likely than $p_{\text{selection}}\left(A\right)$ over a symmetric distribution.

Perhaps a simpler takeaway from these $\ne$ bounds is this: if you can find a $U$ so that the values of $\nu$ for $A$ and $B$ are never equal at any point in it’s orbit, then the expectation for $B$ is $k$ times greater than for $A$. That is, if $\nu (A,gU)\ne \nu (B,gU)$ for all $g\in G$, then $k\cdot {\sum }_{g\in G}\nu (A,gU)\le {\sum }_{g\in G}\nu (B,gU)$. If such a $U$ exists, there will be some $G$-symmetric distribution where $k\cdot E_{U\sim D}\left[\nu \right(A,U\left)\right]\le E_{U\sim D}\left[\nu \right(B,U\left)\right]$, though it need not hold for all of them.

7. Discussion

7.1 The Weak Natural Abstraction Principle

A (much) weaker version of the natural abstraction hypothesis would be just: “some abstractions are usually preferred to others” [by most agents, in natural environments]. In this form it is obviously true; if nothing else, surely we can come up with some abstractions which are quite useless, and some which are not. As we advanced through the post, we emphasized how we weren’t yet able to guarantee strong convergence of abstractions, such that most agents prefer only a “small” number of natural ones. However, we can also see the theorems of this post as providing gradually stronger demonstrations of this weak natural abstraction principle (for our setting).

Already from the original Blackwell Informativeness Theorem, we saw that some information structures are strictly preferred to others (in the sense that ${\phi }^{\prime }{\le }_{BW}\phi$ and ${\phi }^{\prime }{\nleqq }_{BW}\phi$) - those providing a superset of information about the world, and so a superset of attainable outcomes given that information structure. Through the GR and W settings, we saw that if there is non-trivial structure to the environment’s “outcomes” (which agents are allowed to care about), then information structures need only provide a ‘dominant’ set of information relevant to controlling these outcomes. This still requires enabling a superset of attainable outcomes, but environmental structure makes this relationship more common. Finally, by moving from ‘preferred by all’ to ‘preferred by more’, we saw that the requirement could be softened from ‘a superset’ of outcomes, to ‘unambiguously more’ outcomes (in the sense that we could get a superset up to a permitted relabelling of outcomes).

Even the first type of relationship is guaranteed to exist in any nontrivial environment, and so to guarantee us common preferences: bigger information structures with more information will be preferred to similar smaller ones (as long as both are smaller than the whole environment). But this doesn’t really capture the spirit of the natural abstraction hypothesis. It tells us nothing about preferences between information structures with the same amount of information, or different amounts of distinct information. By the final case, however, we get something more in the right spirit: as long as the environment has at least some information which is more relevant to controlling outcomes than the rest, there will be convergent preferences towards abstractions which capture that information. Many more abstractions can be compared in terms of ‘how much’ policy-relevant information they provide.

We’ve then essentially proven an especially weak (and indeed ‘obvious’) form of the natural abstraction hypothesis. The setting where we prove this is bare-bones, but correspondingly universal: while there are many other elements we’d want to include for a more thorough study of abstractions, basic preferences over ‘ways of coarse-graining information’ will still apply.

Of course, it’s only stronger forms of the hypothesis that would be of much practical use for alignment. Rather than “some abstractions are generally preferred to others”, we’d need something more like “most agents will converge on a ‘small’ number of discrete potential abstractions^[26]”. To have a chance to get a result like this, we would need to introduce some of those missing elements.

7.2 Weak natural abstraction as power-seeking

Intuitively, there’s a clear connection between natural abstraction and instrumental convergence: where natural abstraction proposes that certain ‘abstractions’ are more useful to a wide range of agents (in a given environment), instrumental convergence proposes that certain states or actions are more preferred by a wide range of agents. Since here we have a formal working notion of ‘abstractions’, as information structures, we can make this connection more explicit. Indeed, in our power-seeking-theorem style analysis, we basically already have—but let’s see directly how convergent preferences over abstractions can be seen as a special case of power seeking.

Recall that ‘power’ in the sense of these theorems is a concept about the choices available to you, so ‘seeking power’ only makes sense if you are making at least two successive choices. We’re then looking for a sequence of choices which occur in our setting—which are, naturally, 1) your choice of information structure $\phi$, 2) your choice of policy $\pi$ given the information structure.

We can choose to view this as an a two-timestep MDP, with choice of $\phi$ leading to a dedicated state $s_{\phi }$, and choice of $\pi$ leading stochastically to an outcome $w\in W$ via $T$. Intrinsic reward for each information state $s_{\phi }$ is 0, and the reward of an outcome is it’s utility^[27].

Then, the standard power-seeking results applied to this MDP^[28] tell us that most parametrically retargetable decision-makers (including but not limited agents choosing according to some utility function) will ‘tend’ to choose information structures which give them ‘more’ attainable outcomes via their choice of policy, and hence have higher ‘power’: higher average optimal value^[29] attainable from that state, averaging over utility functions.

The original versions of these theorems give us this ‘tendency’ in a slightly different form than we’ve used here (“a larger number of orbit elements of a decision-making function which strictly prefer one information structure vs. another”). As discussed earlier, we deliberately use a different formulation to ensure our tendencies (common preferences) are transitive. But up to this reformulation, which could equally be applied to the general power-seeking context, these results are exactly the same ones we saw in section 6.

We can then say that “weak natural abstraction [exists, and] is equivalent to power seeking tendencies in the agent’s choice of abstraction”, and prove it in a particular formal sense. Tendencies towards preferring certain (natural) abstractions occur exactly when instrumental convergence occurs in an information-selection problem derived from the environment.

7.3 Limitations and future work

We’ve built up one step at a time, from the minimal possible setting and the simplest questions. As a result, we’ve so far ignored many key aspects of realistic abstractions. We’ll briefly go over a few of the most important missing pieces of the story, and how they relate to this framework.

1. The first missing piece can be summarized as: we ignore “internal” and “local” structure of all kinds. We treat the environment, signal, actions, and outcomes as undifferentiated sets—we don’t factor them into variables representing e.g. different parts of the environment, and relate such structure to the mappings between objects. Relatedly, we don’t consider any notion of ‘similarity’ between different world states, signals, or objects. As an important special case, this includes a lack of any real temporal structure.

   The real ‘substance’ of the natural abstraction hypothesis is about the structure of the environment. Local structure is key to stories of why natural abstractions should exist, and what kind of thing they should be—for example, “information relevant to ‘far away’ variables” inherently requires some notion of distance (usually, a distance in a causal graph). This is well motivated by the local causal structure of the real world. But local structure in the signal and the encoder may also play an important role. For instance: this is what’s is required to represent ideas like “similar world states are encoded by similar signals” or “a particular ‘part’ of the signal always corresponds to a particular ‘part’ of the environment”. Moreover, the signal and encoder will also be embodied in the physical world. I expect such structure may be necessary to properly understand e.g. the modularity of abstractions.

   Introducing internal structure to the environment is the main bridge to Wentworth’s work on abstractions, and an obvious next step^[30]. Introducing internal structure to the agent and it’s boundaries is less-explored, but likely important. This is also where we would model distributions over functional behavior pushed forward from distributions over parameters, via some parameter-function map.

1. Second, we touch only very lightly on similarities between abstractions. Of course, we should care not only whether different agents use exactly the same information structures, but also if they use ‘similar’ ones. While internal structure opens up more options, we could look at extrinsic similarities even using only the current setting. ‘Contains a superset of the information’ is one relationship of this type, and we mention in passing ‘orthogonality’ of deterministic information structures, but there are many more possibilities (starting from mutual information $I\left(\phi \right(X);{\phi }^{\prime }(X\left)\right)$ - but this alone doesn’t constrain ‘what’ information is shared).

2. Third, we’ve ignored all computational aspects of the problem—and thereby any ‘reasoning’. The message-size of an abstraction can be considered a particular kind of space complexity, but there are many other tradeoffs—we may be concerned with the the algorithmic (or circuit, etc.) complexity of encoders and policies, as well as of the problems of selecting them. In principle all of this could be cast in terms of cost functions on internal structure, but is also well worth considering from an abstract computational perspective.

3. Finally, we assumed throughout the optimal choice of policy; and towards the end, we considered optimality with respect to a set of information structures. These can (and should) be generalized from optimality to other criteria. Some of this generalization can be transferred directly from the parametric retargetability setting with a little more care (though there are some details to watch to maintain transitivity). While part of the motivation for non-optimal choice can be addressed through appropriate costs on structure and computation, it is also simply worth generalizing beyond deterministic selection.

A convenient property of our setting is that it’s fairly clear how one could add all of these pieces.

7.4 Additional remarks

A few other miscellaneous remarks:

• Adding a lossy ‘perception’ between $X$ and $\phi$ (i.e. $X\stackrel{\rho }{\to }O\stackrel{\phi }{\to }M$) so that $\phi$ doesn’t have direct access to the environment does nothing more, nor less, than restricting the space of possible information structures. Namely, it restricts to those of the form $\gamma \rho$, a.k.a. those in the cone $\le \rho$.

• Transferring some of the ideas here back to the general power-seeking context, we can observe for example that a) while no individual action strictly dominates any other, some sets of actions can strictly dominate other (even disjoint) sets, b) a state $s_{\alpha }$ can be strictly preferred to $s_{\beta }$ not just over all distributions symmetric under the permutation of every state, but even over all distributions (or $U$) symmetric only under interchanging the values of $s_{\alpha }$ and $s_{\beta }$.

• Another way to view this whole setting is as a problem of lossy compression, which we can analyze through rate-distortion theory. Our interest is different from the usual applications rate-distortion theory in that we’re interested in commonalities between the encoders for broad families of abstract “distortions” (derived from utility functions) which may not have anything to do with ‘reconstruction’, as well as that we currently consider only a single timestep. But similarities among distortion-rate functions for different distortions, especially e.g. phase changes at consistent capacities, may certainly be relevant.

8. Conclusion

We started with a simple formal model of abstraction, where an agent must coarse-grain information about a larger environment into a smaller number of internal states. In this model, we asked: when do different agents prefer to abstract in the same way?

We found that while there are few preferences common among all general-form utility functions, as we add more structure to their domain, we can get proportionally stronger convergence. When restricted to preferences over a shared environment, all agents may prefer exactly the same abstractions—or indeed completely different ones, depending on the environment. We gave examples of several possible convergence behaviors, including cases where abstractions converge strongly at particular capacities, but are orthogonal across capacities. We then generalized these results, using a modified version of Turner’s power-seeking formalism to show that many more preferences are shared by ‘most’ utility functions, under priors obeying certain symmetries. Based on this, we showed how natural abstractions can formally be viewed as a type of power-seeking.

Some high-level takeaways:

• For agents facing the same environment, there is indeed almost always some convergence in preferred abstractions. How much depends on the environment, and the relationships it creates between the attainable outcomes using different abstractions.

• Even if an environment leads to convergent abstractions, there’s no general reason why it can’t lead to unrelated abstractions for agents with different capacities. Quite likely, the same is true for other differences in constraints. There are arguments^[14:1] why this might not a problem, and there may be conditions which prevent this from being an obstacle in practice, but we so far lack a formal model of this.

And a meta takeaway: taking the perspective that ‘abstractions are ways of coarse graining the environment’ literally and building a straightforward model of it gives a very tractable setting, with connections to many established fields. By default it lacks key features we’d want to fully model abstractions, but it comes with handles to introduce several of them.

Overall, even if the strongest (and most convenient) forms of the natural abstraction hypothesis turn out to be false, there will surely be some degree of convergence among the abstractions used by different agents in practice. Understanding the factors behind this convergence, and behind the formation of abstractions generally, would be useful in order to know what to expect of AI systems. Even short of strong natural abstractions, these dynamics inform the viability of different interpretability and alignment techniques—under what conditions we can trust them, and when we can expect them to generalize. Furthermore, they have multiple connections to the dynamics of which goals agents will acquire (i.e. agent-selection-processes will bring about) in which circumstances—of obvious importance.

While the theory here is only a very early step, we hope it can help clarify some of the broadest strokes, and show directions for further development.

---

Appendix: Omitted Proofs

Proof of “$g$-injection of outcomes implies EU-dominated under $G$-invariant distributions”

Let $\nu$ be a function which scores a set of attainable outcome-distributions against a utility function, so $\nu :P\left(R^{\left|W\right|}\right)\times R^{\left|W\right|}\to R$, with the following properties:

1. Monotonicity: if $A\subseteq B\subseteq R^{\left|W\right|}$, then $\nu (A,U)\le \nu (B,U)$ for all $U$.

2. $G$-invariance: for all $A\subseteq R^{\left|W\right|}$, $U\in R^{\left|W\right|}$ and $g\in G$, we have $\nu (g\cdot A,g\cdot U)=\nu (A,U)$

Then $\nu$ also naturally scores information structures, by $\nu (\phi ,U)=\nu \left(D\right(\phi ),U)$.

Lemma: If $g^{\ast }\cdot A\subseteq B$, then ${\sum }_G\nu (A,gU)\le {\sum }_G\nu (B,gU)$.

Proof: We have ${\sum }_G\nu (A,gU)={\sum }_G\nu (g^{\ast }A,g^{\ast }gU)={\sum }_G\nu (g^{\ast }A,gU)\le {\sum }_G\nu (B,gU)$, with the three equalities following from $G$-invariance, the sum being over the same orbit, and monotonicity respectively. $\square$

Corollary: For any $G$-invariant distribution $D$ over utility functions, $E_D[\nu (A,U)]\le E_D[\nu (B,U)]$.

Proof: Since each element of the orbit $\{gU:g\in G\}$ of any $U\in D$ has the same density ${\mu }_D\left(gU\right)={\mu }_D\left(U\right)$, the expectation is unchanged if we replace $\nu (A,U)$ by its average over the orbit $\frac{{\sum }_G\nu (A,gU)}{\left|G\right|}$: $$E_D[\nu (A,U)]={\int }_D\nu (A,U)d{\mu }_D\left(U\right)={\int }_D\frac{{\sum }_G\nu (A,gU)}{\left|G\right|}d{\mu }_D\left(U\right)$$ The lemma says that $\frac{{\sum }_G\nu (A,gU)}{\left|G\right|}\le \frac{{\sum }_G\nu (B,gU)}{\left|G\right|}$ at each point, so $E_D[\nu (A,U)]\le E_D[\nu (A,U)]$ (and in fact the converse also holds: if this is true for all $G$-invariant $D$, then we must have ${\sum }_G\nu (A,gU)\le {\sum }_G\nu (B,gU)$ for all $U$.) $\square$

Then to complete the theorem, we just need to confirm that $\nu (\phi ,U)=U\left(\phi \right)={max}_{d\in D\left(\phi \right)}U\left(d\right)$ is monotonic and $G$-invariant for any $G$, and so is the optimality of $\phi$ out of a $G$-symmetric set of options^[31] $\Phi =G\cdot \Phi$: ${\mathrm{IsOpt}}_{\Phi }(\phi ,U)$, which are easily checked. $■$

---

1. ↩︎

   Specifically, in terms of theorems,

   • The Blackwell Informativeness Theorem and Good Regulator Theorem are of course far from novel

   • Somewhat to my surprise, I haven’t been able to find the “GR-Blackwell Order” result anywhere. Still, I wouldn’t be shocked if an equivalent result had come up in some other context. The connection appears to be new—there seem to be no sources at all which discuss both the Blackwell order and the Good Regulator Theorem.

   • The power-seeking + Blackwell results I expect to be novel. The modified formulation of the power-seeking results is also novel.

2. ↩︎

   And more confidently: we can at least say that this is an important facet of the agent’s abstractions.

3. ↩︎

   Suppose $\phi \left(x_0\right)={\lambda }_i{m}_i+{\lambda }_j{m}_j+{\sum }_{k\ne i,j}{\lambda }_k{m}_k$, with $0<{\lambda }_i,{\lambda }_j<1$ (since ${\sum }_k{\lambda }_k=1$, if one coefficient is strictly in $(0,1)$ then two are). Let $\pi$ be the optimal policy for $\phi$. If $U(x_0,\pi (m_i\left)\right)<U(x_0,\pi (m_j\left)\right)$, then ${\phi }^{\prime }$ with ${\phi }^{\prime }\left(x_0\right)=({\lambda }_i+{\lambda }_j)m_j+{\sum }_{k\ne i,j}{\lambda }_k{m}_k$ does at least as well; if $U(x_0,\pi (m_i\left)\right)\ge U(x_0,\pi (m_j\left)\right)$ then ${\phi }^{\prime }\left(x_0\right)=({\lambda }_i+{\lambda }_j)m_i+{\sum }_{k\ne i,j}{\lambda }_k{m}_k$ does. This has reduced the number of coefficients in $(0,1)$ by at least 1; repeat until there are none. If any of these $U$-inequalities are strict, than ${\phi }^{\prime }$ is strictly preferred. Suppose they are all equalities; if $\pi \left(m_i\right)=\pi \left(m_j\right)$ for any $i\ne j$, then we can combine those signals and get a smaller ${\phi }^{\prime }$; if not, $U(x_0,\pi (m_i\left)\right)=U(x_0,\pi (m_j\left)\right)$ means that $U$ is indifferent between distinct outcomes.

4. ↩︎

   And again, any fixed nondegenerate prior $p\left(x\right)$ would also work.

5. ↩︎

   We use two types of diagrams in this post. The first is the “graphical model” style (as e.g. in section 1), and the second is commutative diagrams (as here). We mention this because the same picture would mean something completely different as a graphical model vs. as a commutative diagram. We use boxes around the nodes in graphical models, and no boxes in commutative diagrams.

6. ↩︎

   In other words, each state has positive measure for some some $x$: $\forall s\in S:\exists x\in X:[f{]}_{xs}=p_f(s|x)>0$

7. ↩︎

   To be a little bit more explicit about how the ‘multiple agents’ vs. ‘multiple games’ interpretations relate here:

   • ″$\phi$ is preferred for all $U$” $⟺$ ”$\phi$ is preferred for (all individual agents facing) arbitrary distributions over $U$”

   • ″$\phi$ is optimal for all $U$” $⟺$ ”$\phi$ is optimal for some distribution over $U$ with full support” (for optimality with respect to any upward-closed universe of information structures).

   • However, $\phi$ may be preferred by a particular distribution with full support without being preferred by all $U$, and $\phi$ may be optimal for the distribution with respect to a non-closed set of information structures without being optimal for any $U$.

8. ↩︎

   We use ‘all tasks’ instead of the Gooder Regulator Theorem’s ‘a sufficiently rich set of tasks’ - which turns out to mean exactly ‘a set of tasks equivalent to all tasks for the purposes of this system’. As a side effect of our re-proof of GRT, we’ll be able to get the corollaries a) such a set of tasks exists, b) for a set of tasks to be sufficiently rich for all possible systems, it must be dense in the set of all tasks, c) for a single system, $|X_S{|}^2$ tasks suffice (a loose bound).

9. ↩︎

   This follows from the fact that column normalization is equivalent to multiplication on the right by an appropriate diagonal matrix, which is invertible.

10. ↩︎

    Picking any (consistent) arbitrary value of the $\mathrm{argmax}$ if there is more than one. We will assume this throughout.

11. ↩︎

    Consider $x_i$ which all map onto a 1-dimensional line $\alpha f\left(x_1\right)+(1-\alpha )\left(f\right(x_k\left)\right)$ - merging any number of pairs of adjacent states gives a Pareto partition for its size, and these are the only ones. For illustration: let $f\left(x_1\right)=(0.5,0.25,0.25),f\left(x_2\right)=(0.5,0.30,0.20),f\left(x_3\right)=(0.5,0.40,0.10)$ be distributions over $S=\{s_1,s_2,s_3\}$); then $\left\{\right\{x_1,x_2\},\{x_3\left\}\right\}$ and $\left\{\right\{x_1\},\{x_2,x_3\left\}\right\}$ are Pareto optimal, but $\left\{\right\{x_1,x_3\},\{x_2\left\}\right\}$ is not. This is the smallest possible number of Pareto partitions for a given $|X_S|$.

12. ↩︎

    N.b. we say ‘world’ rather than ‘world model’, because this defines the dynamics of the external environment, not an internal representation the agents might form of those dynamics.

13. ↩︎

    Add an additional dimension $ϵ\in \left[\mathrm{1..100}\right]$, and make only $T(x_{i,\ast ,ϵ>1},a_{1,i,r})=w_r$ (with $T(x_{i,\ast ,1},a_{1,i,r})=w_{\varnothing }$). Then you get a unique family of dominant partitions of size $4$, with the $ϵ>1$ states divided along the first dimension, and the $ϵ=1$ states assigned arbitrarily to those parts. But these partitions are all equivalent to each other under all $U$, so there is still only one dominant equivalence class.

14. ↩︎↩︎

    On the other hand, perhaps we don’t care if differently capable agents ‘use’ the same ‘abstractions’ in the first place, for the senses in this post.

    Wentworth gives the example of DNA: we can recognize and reason about how a tree-DNA-sequence-distribution screens off certain information between specific trees without containing the tree-DNA-sequence-distribution itself in our heads. This demonstrates that the agent need not—simultaneously and non-counterfactually—contain the information of the low-dimensional summary for it to be relevant; the naive ‘size of the abstraction’ isn’t necessarily binding.

    Seems intuitively compelling; does this solve the whole problem? Do we still need to worry about sufficiently large/​complex concepts such that we can’t even store their index, or concepts with insufficiently simple/​factorable implications, or other cases where perhaps this fails?

    Well, we should build some models of how exactly this works and try to find out. So far I have only vague ideas for how to do that, but you’d certainly need many kinds of structure we haven’t yet included.

15. ↩︎

    Rather than e.g. assigning each state utility to be IID uniform on $[0,1]$, as some have used, I think the more natural maximum entropy distribution is to consider utility functions as ‘directions’ (given their equivalence up to affine transformations), and so take the uniform distribution over the $n$-sphere. This turns out to be equivalent to taking each state utility to be I.I.D $N(0,1)$ and renormalizing.

16. ↩︎

    Essentially: even for ${\phi }_1,{\phi }_2,{\phi }_3$ with the same orbit of utilities, the tuples of utility values for different permutations can be aligned like $(1,2,3)$, $(2,3,1)$, $(3,1,2)$, such that each dominates than the previous under most permutations (i.e. is greater at most indexes of the tuples). This follows from a similar pattern in the maximum attainable share of different outcome vectors. As concrete example: let $X=\{1,2,3\}$, $A=\{1,2\}$, $V=(3,2,1,0,0,0)$, and $${\phi }_1=\left(\begin{array}{ccc}1& 0& 0\\ 0& \frac{1}{2}& 0\\ 0& \frac{1}{2}& 1\end{array}\right){\phi }_2=\left(\begin{array}{ccc}0& 0& 1\\ \frac{1}{2}& 0& 0\\ \frac{1}{2}& 1& 0\end{array}\right){\phi }_3=\left(\begin{array}{ccc}0& 1& 0\\ 0& 0& \frac{1}{2}\\ 1& 0& \frac{1}{2}\end{array}\right)$$ with $D$ the uniform distribution over all possible assignments of utilities $v\in V$ to outcomes $(x,a)$ (so that $G=S_6$). A computer evaluation confirms that we then have ${\phi }_1{>}^{!}{\phi }_2{>}^{!}{\phi }_3{>}^{!}{\phi }_1$, but ${\phi }_1{\nless }^{!}{\phi }_2$.

17. ↩︎

    If you like, from $(gu{)}^{T}w=u^T{g}^{-1}w$, where we once again consider $U$ as a vector $u\in R^{|W|}$.

18. ↩︎

    In order for $\Delta W$ to be closed under the group action, and for the action to be linear, it turns out $G$ must act only by permuting the elements of $W$ - this is one reason we must require $G$ to be a subgroup of $S_{\left|W\right|}$ rather than the orthogonal group $O\left(\left|W\right|\right)$.

19. ↩︎

    Compare this to the alternate form for the GR-Blackwell order in section 4.1.1: both involve compositions on the right of $\phi$, but in the GR case we apply a fixed homomorphism $f^T$ before both $\phi$ and ${\phi }^{\prime }$, while here we look for the existence of an isomorphism $g$ asymmetrically.

20. ↩︎

    The elements of $g\cdot D\left(\phi \right)$ are $$\begin{array}{cc}g\sum _{i=1}^{\left|X\right|}\sum _{j=1}^{\left|M\right|}\sum _{k=1}^{\left|A\right|}\left[\phi {]}_{ij}{\pi }_{jk}\right(x_i,a_k)& =\sum _{i=1}^{\left|X\right|}\sum _{j=1}^{\left|M\right|}\sum _{k=1}^{\left|A\right|}\left[\phi {]}_{ij}{\pi }_{jk}g\right(x_i,a_k)\\ & =\sum _{i=1}^{\left|X\right|}\sum _{j=1}^{\left|M\right|}\sum _{k=1}^{\left|A\right|}\left[\phi {]}_{ij}{\pi }_{jk}\right(g{x}_i,g{a}_k)\\ & =\sum _{i=1}^{\left|X\right|}\sum _{j=1}^{\left|M\right|}\sum _{k=1}^{\left|A\right|}\left[\phi {]}_{ij}{\pi }_{jk}^{\prime }\right(g{x}_i,a_k)\\ & =\sum _{i=1}^{\left|X\right|}\sum _{j=1}^{\left|M\right|}\sum _{k=1}^{\left|A\right|}\left[\phi {]}_{ij}{\pi }_{jk}^{\prime }\right(x_{g\left(i\right)},a_k)\end{array}$$ where ${\pi }_{jk}^{\prime }={\pi }_{jg\left(k\right)}$. This is exactly the elements of $D(\phi \circ g)$ - as $\pi \leftrightarrow {\pi }^{\prime }$ is a bijection on the set of all policies, the union over all $\pi$ and over all ${\pi }^{\prime }$ is the same.

21. ↩︎

    In the case where $G=S_{|X\times A|}$, restricting to deterministic $P,P^{\prime }$, we can show that a) the orbits of certain $U\in \left\{\right(X\times A)\to R\}$ disagree on preferences between some $P,P^{\prime }$, so not everything is equivalent (and also the relation is not a total order), b) if you shift a element from a larger part in the partition to a smaller part, the resulting partition is at least as good—and therefore, a sufficient condition for $P\ge P^{\prime }$ is that the vector of part sizes for $P$ majorizes the vector of part sizes for $P^{\prime }$. This is definitely the kind of thing that can be expressed by some stochastic matrix equation, but I’m not sure what the exact condition is for general $\phi$ - perhaps something like $\gamma \phi \eta ={\phi }^{\prime }$ for column-stochastic $\gamma$ and doubly-stochastic $\eta$. The difference between orderings under independent vs. joint permutations appears to parallel the refinement vs. dominance orders on integer partitions.

22. ↩︎

    I believe it translates to: “Suppose $E_{U\sim D}[U\left({\phi }^{\prime }\right)]\le E_{U\sim D}[U\left(\phi \right)]$ for all $G$-invariant $D$ - then if there exist stochastic ${\gamma }_g$ such that ${\phi }_0={\gamma }_g\circ {\phi }^{\prime }\circ g$ for all $g\in G$, there must also exist stochastic ${\gamma }_g^{\prime }$ and ${\lambda }_g\in [0,1]$, $\sum {\lambda }_g=1$ such that ${\phi }_0={\sum }_G{\lambda }_g{\gamma }_g^{\prime }\phi \circ g$”.

23. ↩︎

    This comes down to the lemma: if $|B\setminus A|/|A\setminus B|\ge c\ge 1$ and $|C\setminus B|/|B\setminus C|\ge c$, then $|C\setminus A|/|A\setminus C|\ge c$. Presumably there is some nice proof of this, but you can also see that it will be true by drawing $A,B,C$ as overlapping collections of line segments, and prove it by solving the constrained minimization of $|C\setminus A|/|A\setminus C|$.

24. ↩︎

    Intuitively it should be possible to weaken the involution condition, which constrains our choice of $G$, but it’s not immediately obvious to me how. Something like “the $g_i$ send $D\left({\phi }^{\prime }\right)\setminus D\left(\phi \right)$ to $D\left(\phi \right)\setminus D\left({\phi }^{\prime }\right)$”, which is one of the things the original conditions accomplish, and with c) we then further have ”$g_i\left(D\right({\phi }^{\prime })\setminus D(\phi \left)\right)$ are disjoint”, but there may be some additional requirement.

25. ↩︎

    This observation, together with the transitivity of ${\le }_{\ne \nu }$, shows that in fact Turner’s orbit-level tendencies are transitive for binary functions.

26. ↩︎

    Or at least a manageable number of manageably-low-dimensional manifolds of potential abstractions.

27. ↩︎

    Writing slightly loosely, $$\begin{array}{cc}& S=\left\{s_0\right\}\cup \{s_{\phi }:\phi \in (X\to \Delta M\left)\right\}\cup \left\{s_{x,a}\right\}\\ & A=\{a_{\phi }:\phi \in (X\to \Delta M\left)\right\}\cup \{a_{\pi }:\pi \in (M\to \Delta A\left)\right\}\\ & T:\\ & (s_0,a_{\phi })\mapsto s_{\phi }\\ & (s_{\phi },a_{\pi })\mapsto T(⟨X,\pi (\phi \left(X\right))⟩)\\ & R\left(s_{\phi }\right)=0\\ & R\left(s_{x,a}\right)=U(x,a)\end{array}$$

28. ↩︎

    To be clear, the current version of these theorems doesn’t require the setting to be an MDP; but an MDP is still needed for the definition of ‘power’ and so for interpretation in terms of ‘power seeking’.

29. ↩︎

    In our setting, a distinction between ‘average optimal value’ and ‘average optimal value, excluding the reward of the state itself’ is irrelevant, as we insist the $s_{\phi }$ whose power we consider have no intrinsic value.

30. ↩︎

    Arguments that ‘most utility functions care only about information relevant far away, because most of the things they might care about are far away’ and ‘most agents care more about information the more copies of it appear in the environment’ have especially clear translations.

31. ↩︎

    As ${max}_{D\left(x\right):x\in \Phi }gU\left(x\right)={max}_{gD\left(x\right):x\in \Phi }U\left(x\right)={max}_{D\left(x\right):x\in \Phi }U\left(x\right)$ - note the $G$ outcome-invariance of $\Phi$ is necessary, otherwise the optimality indicator is not $G$-invariant and we can find counterexamples to the conclusion.

[Nathan Helm-Burger]

So, I think that one of the end goals of seeking to understand natural abstractions is being able to predict what abstractions get chosen given a certain training regime (e.g. the natural world, or a experimental simulation, or a static set of training data). It seems likely to me that a simpler environment, such as an Atari game, or the coin maze, is a good starting point for designing experiments to try to figure this out. I think ideally, we could gradually build up to more complex environments and more complex agents, until we are working in a sim that is usefully similar to the natural world such that the agent we train in that sim could plausibly produce artifacts of cognitive work that would be valuable to society. I try to describe something like this setup in this post: https://​​www.lesswrong.com/​​posts/​​pFzpRePDkbhEj9d5G/​​an-attempt-to-steelman-openai-s-alignment-plan

The main drawback I see to this idea is that it is so vastly more cautious an approach than leading AI labs are currently doing, that it would be a hard pill for them to swallow to have to transition to working only in such a carefully censored simulation world.