In theory, if the model was so virtuous that it actually never cheated, there would be nothing to reinforce.
The smarter models are, the less this is true.
Cheating isn’t really a natural category. It’s just a behavior that gets reward, implied by the structure of the world. There are a continuum of such behaviors (measured by divergence from the pretrained policy). A key facet of smartness is generalizing from small parts of the continuum to other parts and getting it right on the first try.
For models in current RL training, this probably converges to eval awareness and reward-seeking, even just as a way to quickly get intended rewards on environments where no cheating is possible.
Of course, models aren’t just trained by RL on tasks. SFT and RL for prosocial behavior do some amount of work, especially if they’re deliberately pushing back against reward-seeking. But there are gaps we don’t understand in the safety properties here. Humans stubbornly internally represent wireheading as low-predicted-reward despite that being an inaccurate generalization from training. How well, exactly, does RLHF do the same sort of thing, and how do we stop that property from being broken down by other training / other terms in the reward?
The smarter models are, the less this is true.
Cheating isn’t really a natural category. It’s just a behavior that gets reward, implied by the structure of the world. There are a continuum of such behaviors (measured by divergence from the pretrained policy). A key facet of smartness is generalizing from small parts of the continuum to other parts and getting it right on the first try.
For models in current RL training, this probably converges to eval awareness and reward-seeking, even just as a way to quickly get intended rewards on environments where no cheating is possible.
Of course, models aren’t just trained by RL on tasks. SFT and RL for prosocial behavior do some amount of work, especially if they’re deliberately pushing back against reward-seeking. But there are gaps we don’t understand in the safety properties here. Humans stubbornly internally represent wireheading as low-predicted-reward despite that being an inaccurate generalization from training. How well, exactly, does RLHF do the same sort of thing, and how do we stop that property from being broken down by other training / other terms in the reward?