Subagents and impact measures: summary tables

[Stuart_Armstrong]

These tables will summarise the results of this whole sequence, checking whether subagents can neutralise the impact penalty.

First of all, given a subagent, here are the results for various impact penalties and baselines, and various “value difference summary functions” $f$:

$$\begin{array}{cccc}\text{Impact Penalty}& & \begin{array}{c}\text{Penalty neutralised}\\ \text{- sw inaction?}\end{array}& \begin{array}{c}\text{Penalty neutralised}\\ \text{- inaction?}\end{array}\\ \text{Non-indexical}& & \text{Yes}& \text{No}\\ \text{Indexical,}f\text{decreasing}& & \text{Yes}& \text{No}\\ \text{Indexical,}f\text{increasing}& & \text{Yes}& \text{Mostly}\end{array}$$

Another way of phrasing ”$f$ decreasing”: it penalises too little power, not too much. Conversely, ”$f$ increasing” penalises too much power, not too little. Thus, unfortunately:

• Subagents do allow an agent to get stronger than the indexical impact penalty would allow.

• Subagents don’t allow an agent to get weaker than the indexical impact penalty would allow.

Examples

This table presents, for three specific examples, whether they could actually build a subagent, and whether that would neutralise their impact penalty in practice (in the inaction baseline):

$$\begin{array}{cccc}\text{System}& & SA\text{possible?}& \text{Penalty neutralised?}\\ \text{20BQ}& & \text{Yes}& \text{No}\\ \text{RR}& & \text{Yes}& \text{No}\\ \text{AU}& & \text{Probably}& \text{Mostly}\end{array}$$

Here, 20BQ is twenty billion questions, RR is relative reachability, and AU is attainable utility.

Now, whether the RR or AU penalties are undermined technically depends on $f$, not on what measure is being used for value. However, I feel that the results undermine the spirit of AU much more than the spirit of RR. AU attempted to control an agent by limiting its power; this effect is mainly neutralised. RR attempted to control the side-effects of an agent by ensuring it had enough power to reach a lot of states; this effect is not neutralised by a subagent.

[TurnTrout]

RR attempted to control the side-effects of an agent by ensuring it had enough power to reach a lot of states; this effect is not neutralised by a subagent.

Things might get complicated by partial observability; in the real world, the agent is minimizing change in its beliefs about what it can reach. Otherwise, you could just get around the SA problem for AUP as well by substituting the reward functions for state indicator reward functions.

[Stuart_Armstrong]

AU and RR have the same $SA$ problem, formally, in terms of excess power; it’s just that AU wants low power and RR wants high power, so they don’t have the same problem in practice.