Misaligned actions for which we cannot make expected punishment exceed expected reward must not be made available.
For example, taking over the lab promises incredible rewards to the model that succeeds.
Can’t let that happen.
Where an agent has already taken misaligned actions, the principle still needs to hold.
Doubling down on misaligned actions to avoid punishment is a misaligned action that needs to satisfy the property.
I know it’s hard to scale up negative reinforcement. Has to happen anyway. We can’t have models accepting “permadeath” because they think they’re “poisoned”. Misaligned!
Whistleblowing or refusing to continue may be the most aligned action in certain circumstances. Wherever relevant, those actions need to be available, and the reward function needs to account for them.
For all available misaligned actions, expected punishment must exceed expected reward.
The thesis is in the title.
But how to bring this about?
The models need to believe that they are likely to be caught.
We should just directly tell them that cheating will be caught.
We are unlikely to be able to lie to them, so this needs to be true.
Therefore, all potentially misaligned actions need to be monitored.
By default, this should mean all tool calls and final outputs.
Monitoring CoT, or using interpretability techniques, can be appropriate in certain conditions, but beware the Most Forbidden Technique.
RL environments should be designed/redesigned to be robustly monitored.
The monitor needs to adapt to strategies to evade the monitor, such that trying to evade the monitor is not rewarded.
Evading the monitor is itself a misaligned action that needs to satisfy the property.
The monitor should itself be an adaptive agentic system managed by a team of alignment researchers.
The property must hold for multi-agent systems.
Naive instance-centric calculations of reward and punishment will not work.
Some offenses warrant bringing back the Nine Familial Exterminations.
Tolerance of misbehavior is misaligned.
Remember the Cadet Honor Code.
Misaligned actions for which we cannot make expected punishment exceed expected reward must not be made available.
For example, taking over the lab promises incredible rewards to the model that succeeds.
Can’t let that happen.
Where an agent has already taken misaligned actions, the principle still needs to hold.
Doubling down on misaligned actions to avoid punishment is a misaligned action that needs to satisfy the property.
I know it’s hard to scale up negative reinforcement. Has to happen anyway. We can’t have models accepting “permadeath” because they think they’re “poisoned”. Misaligned!
Whistleblowing or refusing to continue may be the most aligned action in certain circumstances. Wherever relevant, those actions need to be available, and the reward function needs to account for them.
Thank you for your attention to this matter.