Reward/​value learning for reinforcement learning

A putative new idea for AI control; index here.

Along with Jan Leike and Laurent Orseau, I’ve been working to formalise many of the issues with AIs learning human values.

I’ll be presenting part of this at NIPS and the whole of it at some later conference. Therefore it seems best to formulate the whole problem in the reinforcement learning formalism. The results can generally be easily reformulated for general systems (including expected utility).

POMDP

A partially observable Markov decision process without reward function (POMDP\R), consists of:

  • a finite set of states ,

  • a finite set of actions ,

  • a finite set of observations ,

  • a transition probability distribution ,

  • a probability distribution over the initial state .

  • an observation probability distribution .

The agent interacts with the environment in cycles: in time step , the environment is in state and the agent chooses an action . Subsequently the environment transitions to a new state drawn from the distribution and the agent then receives an observation drawn from the distribution . The underlying states and are not directly observed by the agent.

An observed history is a sequence of actions and observations. We denote the set of all observed histories of length with .

For a given horizon , call the set of full histories; then is the set of partial histories. For , let be the sequence of actions , let be the sequence of observations , and let the sequence of states .

The set is the set of policies, functions mapping histories to probability distributions over actions. Given a policy and environment , we get a probability distribution over histories:

The expectation with respect to the distributions and is denoted .

Rewards

Rewards in this case can be seen as functions from observations to real numbers.

The agent’s goal is to maximize total reward up to the horizon . We assume that and are known to the agent. The reward function is unknown, but there is a finite set of candidate reward functions, . The agent has to learn a reward function in the process of interacting with the environment.

The reward learning posterior

There are a variety of algorithms that act as reward function learning processes. It might be the cooperative learning algorithm, or some interactive question and answers sessions, or simply learning from observation of human behaviour/​human generated data. In all cases, at the end of turns, the agent will have an estimate of the probability of the various reward functions.

Thus a universal definition of the process of reward learning is given by a posterior , mapping histories to distributions over possible rewards. This posterior is equivalent with the definition of the algorithm.

Now, anything that gives a distribution over can therefore give a distribution over .

This allows the construction a value function for any policy corresponding to the reward learning posterior:

The reward learning prior

Some reward learning algorithms (though not all) will also have a reward learning prior over . Given a partial history with , this gives the agent’s current estimate as to what the final distribution over will be: .

For consistency, when , set (so that when all the history is in, the prior is the posterior).

This prior is often used in practice to estimate the value function .