You want to learn an

embedding of the opportunitiesyou have in a given state (or for a given state-action), rather than just its potential rewards. Rewards are too sparse of a signal.More formally, let’s say instead of the Q function, we consider what I would call the Hope function: which given a state-action pair (s, a), gives you a distribution over states it expects to visit, weighted by the rewards it will get. This can still be phrased using the Bellman equation:

Hope(s, a) = rs’ + f Hope(s’, a’)

The “successor representation” is somewhat close to this. It encodes the distribution over future states a partcular policy expects to visit from a particular starting state, and can be learned via the Bellman equation / TD learning.

Left the following comment on the blog: