The Most Forbidden Technique doesn’t/shouldn’t apply to actions taken in the process of achieving a goal. You definitely should be both monitoring (come on guys) and training on tool calls.
In Zvi’s formulation,
“The Most Forbidden Technique is training AI using interpretability techniques.
An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is up to. You could train on that. Never do that. You train on [X]. Only [X]. Never [M], never [T].
Why? Because [T] is how you figure out when the model is misbehaving.
If you train on [T], you are training the AI to obfuscate its thinking, and defeat [T]. You will rapidly lose your ability to know what is going on, in exactly the ways you most need to know what is going on.
Those bits of optimization pressure from [T] are precious. Use them wisely.”
The means that an agent uses to achieve its ends are definitely part of X and fair game for training against! If your reward function doesn’t care whether felonies were committed in pursuit of the goal, you’re doing it wrong.
I have no special insight into OpenAI, but I wonder whether an overbroad understanding of the Most Forbidden Technique isn’t partially behind their baffling failure to monitor not only the CoT (a defensible position) but also the illegal, misaligned actions.
The Most Forbidden Technique doesn’t/shouldn’t apply to actions taken in the process of achieving a goal. You definitely should be both monitoring (come on guys) and training on tool calls.
In Zvi’s formulation,
“The Most Forbidden Technique is training AI using interpretability techniques.
An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is up to. You could train on that. Never do that.
You train on [X]. Only [X]. Never [M], never [T].
Why? Because [T] is how you figure out when the model is misbehaving.
If you train on [T], you are training the AI to obfuscate its thinking, and defeat [T]. You will rapidly lose your ability to know what is going on, in exactly the ways you most need to know what is going on.
Those bits of optimization pressure from [T] are precious. Use them wisely.”
The means that an agent uses to achieve its ends are definitely part of X and fair game for training against! If your reward function doesn’t care whether felonies were committed in pursuit of the goal, you’re doing it wrong.
I have no special insight into OpenAI, but I wonder whether an overbroad understanding of the Most Forbidden Technique isn’t partially behind their baffling failure to monitor not only the CoT (a defensible position) but also the illegal, misaligned actions.