Because motivations are so underdetermined by the reward signal, we may be able to shape them without running into the downstream problems of blocking access to high-rewards, such as deceptive alignment
Could you help me understand this one a bit better? I would have thought that the degenerate mapping of motivations to actions means that many motivations can be mapped onto the same high-reward actions, and deceptive alignment is a case where actions consistently achieve a high reward, but its underlying motivations are misaligned, not vice versa. Sorry if I’m misunderstanding this sentence!
You’re right that deceptive alignment involves misaligned motivations producing high-reward actions. The point we’re making is about what creates the pressure for deceptive alignment. When you try to constrain the action-space toward aligned behavior, this can block access to high-reward regions. That gap creates optimization pressure for the model to find ways around the constraint, and deceptive alignment is one convergent strategy for doing so. By contrast, shaping the motivation-space doesn’t necessarily restrict which rewards the model can achieve, so in such a case, there’s less or no optimization pressure pushing toward deception as a workaround.
Ideally, we want a well-aligned HHH assistant to be strongly motivated to do well in RL reasoning training. Generally, the “helpful” element of that is a good start. The problem is either tasks that are reward-hackable, where doing so at maximal effectiveness is incompatible with “honest”, or any tasks that were morally dubious enough to be problematic from a “harmlessness” point of view.
If I were designing reasoning training tasks for a frontier lab, I would take the time to ensure that they were all plausibly framed in ways that were compatible with an HHH assistant persona working hard on them, at least via helpfulness, and ideally that at least most of them looked like tasks that were genuinely worthwhile and going to help make the world a better place in some way.
Could you help me understand this one a bit better? I would have thought that the degenerate mapping of motivations to actions means that many motivations can be mapped onto the same high-reward actions, and deceptive alignment is a case where actions consistently achieve a high reward, but its underlying motivations are misaligned, not vice versa. Sorry if I’m misunderstanding this sentence!
You’re right that deceptive alignment involves misaligned motivations producing high-reward actions. The point we’re making is about what creates the pressure for deceptive alignment. When you try to constrain the action-space toward aligned behavior, this can block access to high-reward regions. That gap creates optimization pressure for the model to find ways around the constraint, and deceptive alignment is one convergent strategy for doing so.
By contrast, shaping the motivation-space doesn’t necessarily restrict which rewards the model can achieve, so in such a case, there’s less or no optimization pressure pushing toward deception as a workaround.
Ideally, we want a well-aligned HHH assistant to be strongly motivated to do well in RL reasoning training. Generally, the “helpful” element of that is a good start. The problem is either tasks that are reward-hackable, where doing so at maximal effectiveness is incompatible with “honest”, or any tasks that were morally dubious enough to be problematic from a “harmlessness” point of view.
If I were designing reasoning training tasks for a frontier lab, I would take the time to ensure that they were all plausibly framed in ways that were compatible with an HHH assistant persona working hard on them, at least via helpfulness, and ideally that at least most of them looked like tasks that were genuinely worthwhile and going to help make the world a better place in some way.