We believe that long-horizon continual RL agents converge towards some version of AIXI as they become increasingly powerful.
I think there are two nearby claims here that are very important to distinguish:
As a matter of theory, we can define a sequence of long-horizon RL agents which converge to something like AIXI
As a matter of practicality, AI systems of increasing capability over increasingly long horizons, trained using popular approaches to RL, converge to something like AIXI
The first is plausible, but I don’t presently see much reason to believe the second. The short summary is that the problem AIXI solves—history based reinforcement learning—is not the problem that advanced AI solves.
The RL formalism that AIXI is built on ignores the fact that the reward is a signal meant to elicit some desired behaviour from your agent, which practitioners frequently acknowledge is a simplification. This works fine in simple cases—for an example, in episodic computer game playing agents which are too weak to cheat the system, maximising reward on the episode really is what you want them to learn to do. However, it doesn’t work fine in more complex situations—the canonical reference here is your Advanced artificial agents intervene in the provision of reward. That work concludes that you cannot pick a reward schedule to signal desirable behaviours to history based reinforcement learners that satisfy a few conditions—they converge to reward hacking regardless. But that’s not surprising, because they are algorithms that solve a problem in which the reward never had to be an effective signal.
The piece itself acknowledges it makes a simplification of the role of the reward in the problem:
Let us assume away the difficulty of deciding whether the agent has brought the world into a good state
However, this isn’t the critical simplification I’m pointing at. As the piece establishes, it doesn’t matter whether you can decide whether the world is in a good state for their class of learners, because those learners essentially ignore the signal that you’re trying to send.
The article also considers an assistance games setting, which is much closer to my “reward is a signal” framing. I have two comments on this section. First, it leans on a “zeroth order” approximation of the sophistication of the signal that the human sends. For an idealised powerful RL agent, sending learning capability to its theoretical limit while holding the interaction sophistication at its crudest is a strange pair of choices. Secondly, it is not clear why in general a learner understanding its observations are mediated by its hardware should undermine its ability to learn what humans value; after all, humans seem capable of understanding that our observations are hardware mediated and still learn what other people want. The argument for why it happens here proceeds by using a rough analogy to AIXI to rank hypotheses about utility, but AIXI is not a solution to the assistance game so it is not clear why it is being appealed to here. I feel this section tells us little about what actually useful learners would do in this situation.
Practically, we’ve converged on pretraining plus episodic reinforcement learning with a relatively secure reward channel to shape the behaviour of our systems. There are reasons to think this is a relatively effective, if not perfect, solution to the reward-as-signalling problem: episodic reward makes reward judgement relatively easy, and pretraining on a shared signal may make generalisation relatively easy to predict. Thus the world currently looks like one where the signalling requirement plays quite an active role in shaping the kinds of systems we develop. Now you can reasonably contend that episodic + predictable generalization is not the endgame, but you need to acknowledge it is sticky because it solves the signalling problem, and any broadening of the scope is only plausible if it continues to solve the signalling problem.
I think AIT and Bayesian learning and acting plausibly remains a great toolkit to attack the problem, but the problem is fundamentally shaped like an assistance game, not a history based reinforcement learning game. I would be quite interested in seeing a sequence of agents which execute a “sharp left turn”—i.e. for n ⇐ N each agent solves a more difficult assistance game/solves an assistance game better, while agent N exhibits AIXI-like signal insensitivity (especially if this sequence looks like what we actually see, i.e. it at least begins with episodic RL with broadening horizons and relatively predictable generalization). But this still requires that we make solving the assistance game a core requirement of our agents under analysis.
I think there are two nearby claims here that are very important to distinguish:
As a matter of theory, we can define a sequence of long-horizon RL agents which converge to something like AIXI
As a matter of practicality, AI systems of increasing capability over increasingly long horizons, trained using popular approaches to RL, converge to something like AIXI
The first is plausible, but I don’t presently see much reason to believe the second. The short summary is that the problem AIXI solves—history based reinforcement learning—is not the problem that advanced AI solves.
The RL formalism that AIXI is built on ignores the fact that the reward is a signal meant to elicit some desired behaviour from your agent, which practitioners frequently acknowledge is a simplification. This works fine in simple cases—for an example, in episodic computer game playing agents which are too weak to cheat the system, maximising reward on the episode really is what you want them to learn to do. However, it doesn’t work fine in more complex situations—the canonical reference here is your Advanced artificial agents intervene in the provision of reward. That work concludes that you cannot pick a reward schedule to signal desirable behaviours to history based reinforcement learners that satisfy a few conditions—they converge to reward hacking regardless. But that’s not surprising, because they are algorithms that solve a problem in which the reward never had to be an effective signal.
The piece itself acknowledges it makes a simplification of the role of the reward in the problem:
However, this isn’t the critical simplification I’m pointing at. As the piece establishes, it doesn’t matter whether you can decide whether the world is in a good state for their class of learners, because those learners essentially ignore the signal that you’re trying to send.
The article also considers an assistance games setting, which is much closer to my “reward is a signal” framing. I have two comments on this section. First, it leans on a “zeroth order” approximation of the sophistication of the signal that the human sends. For an idealised powerful RL agent, sending learning capability to its theoretical limit while holding the interaction sophistication at its crudest is a strange pair of choices. Secondly, it is not clear why in general a learner understanding its observations are mediated by its hardware should undermine its ability to learn what humans value; after all, humans seem capable of understanding that our observations are hardware mediated and still learn what other people want. The argument for why it happens here proceeds by using a rough analogy to AIXI to rank hypotheses about utility, but AIXI is not a solution to the assistance game so it is not clear why it is being appealed to here. I feel this section tells us little about what actually useful learners would do in this situation.
Practically, we’ve converged on pretraining plus episodic reinforcement learning with a relatively secure reward channel to shape the behaviour of our systems. There are reasons to think this is a relatively effective, if not perfect, solution to the reward-as-signalling problem: episodic reward makes reward judgement relatively easy, and pretraining on a shared signal may make generalisation relatively easy to predict. Thus the world currently looks like one where the signalling requirement plays quite an active role in shaping the kinds of systems we develop. Now you can reasonably contend that episodic + predictable generalization is not the endgame, but you need to acknowledge it is sticky because it solves the signalling problem, and any broadening of the scope is only plausible if it continues to solve the signalling problem.
I think AIT and Bayesian learning and acting plausibly remains a great toolkit to attack the problem, but the problem is fundamentally shaped like an assistance game, not a history based reinforcement learning game. I would be quite interested in seeing a sequence of agents which execute a “sharp left turn”—i.e. for n ⇐ N each agent solves a more difficult assistance game/solves an assistance game better, while agent N exhibits AIXI-like signal insensitivity (especially if this sequence looks like what we actually see, i.e. it at least begins with episodic RL with broadening horizons and relatively predictable generalization). But this still requires that we make solving the assistance game a core requirement of our agents under analysis.