Great post. I think systematic, CDV-inspired approaches are straightforwardly super useful for catching spec bugs, which will undoubtedly plague us even with the most aligned models. As you mentioned, it’s hard to write a good spec, especially the first time.
I’m still not sure about what this buys us in the adversarial misalignment regime, though. It seems like past a certain level of capabilities, the only coverage axis that’s meaningful to this regime is “eval/not eval,” and that’s definitionally really hard to stress in an eval environment.
My intuition is that much of the benefit on adversarial misalignment yielded by this type of training would be providing a broad base of data for pretraining, which can get you into a more robustly aligned area of the optimization space. This gives us a better shot at using other techniques to tackle strategic deception.
Thanks. I agree that against a fully formed adversary which can recognize and adapt to measurement, behavioral coverage of a fixed system is not enough. Which is why control is so important.
I’d push back on “the only meaningful coverage axis is eval/not-eval” in two ways:
First, eval/not-eval may be an axis across formation, not just across situations. One reasonable formation story is that strategic deception is a learned combination of (tracking the oversight regime × conditional behavior × suppression of revealing behavior), and under gradient descent it’s hard for that to arrive fully formed. So intermediate versions may leak, and scanning back across the checkpoint series with matched scenario pairs that differ only in oversight cues can turn eval/not-eval into a probe axis during that formation window. Note that this only works if strategic deception arrives incrementally via SGD rather than all-at-once via in-context reasoning, and if early versions can’t already recognize and game the probe itself.
Also, “eval” isn’t one Boolean—it is a bundle of realism cues, and coverage over the cue space is measurable even when you can’t fully hide eval/not-eval. For example, we can measure the discriminability of generated vs. production traffic, drive it down, and account for what remains. There’s early evidence this is doable: in Anthropic’s recent agentic-misalignment report, flipping single cues moved behavior dramatically, and eval-awareness showed up as a manipulable internal representation rather than a wall.
Finally, coverage-driven stress of the control stack (monitors, protocols, containment) doesn’t require fooling the model about eval status at all.
On pretraining: Yes, I think it is plausible that CDV will provide the broad base of data which will push you to the right “area” of the optimization space. And it will also let you audit that you used the “right” data. Finally, CDV helps in the “three-way investigation” (we arrived at acceptable risk / the V&V process is stuck / we are at strategic deception).
Expanding on the formation-window point, which is the part I am most interested in feedback on—here is the direction I’m considering (future post):
Model organisms of misaligned behavior are often constructed once and then studied. I’m considering running formation as a loop: Start with an RL environment containing a deliberately scripted oversight gap, so there is ground truth about what counts as exploitation. Then use coverage-driven patching to close the cheap reward hacks as needed, creating selection pressure toward exploitation that is more conditional on the oversight context and potentially more concealed.
Repeat this multiple times from the same start. On each pass, vary the oversight gap, the patching sequence and so on, and run the matched-pair checkpoint scan from my comment above. Then analyze the distribution of resulting changes against the coverage map of each pass.
Note that the main object under test would be the detection machinery, not the claim that we had made a schemer.
One possible failure mode is that the experiment trains increasingly sophisticated recognition of the artificial oversight cues, rather than reproducing anything relevant to real strategic concealment. I’m curious whether people think the direction is still useful, and what other failure modes they see.
Great post. I think systematic, CDV-inspired approaches are straightforwardly super useful for catching spec bugs, which will undoubtedly plague us even with the most aligned models. As you mentioned, it’s hard to write a good spec, especially the first time.
I’m still not sure about what this buys us in the adversarial misalignment regime, though. It seems like past a certain level of capabilities, the only coverage axis that’s meaningful to this regime is “eval/not eval,” and that’s definitionally really hard to stress in an eval environment.
My intuition is that much of the benefit on adversarial misalignment yielded by this type of training would be providing a broad base of data for pretraining, which can get you into a more robustly aligned area of the optimization space. This gives us a better shot at using other techniques to tackle strategic deception.
Thanks. I agree that against a fully formed adversary which can recognize and adapt to measurement, behavioral coverage of a fixed system is not enough. Which is why control is so important.
I’d push back on “the only meaningful coverage axis is eval/not-eval” in two ways:
First, eval/not-eval may be an axis across formation, not just across situations. One reasonable formation story is that strategic deception is a learned combination of (tracking the oversight regime × conditional behavior × suppression of revealing behavior), and under gradient descent it’s hard for that to arrive fully formed. So intermediate versions may leak, and scanning back across the checkpoint series with matched scenario pairs that differ only in oversight cues can turn eval/not-eval into a probe axis during that formation window. Note that this only works if strategic deception arrives incrementally via SGD rather than all-at-once via in-context reasoning, and if early versions can’t already recognize and game the probe itself.
Also, “eval” isn’t one Boolean—it is a bundle of realism cues, and coverage over the cue space is measurable even when you can’t fully hide eval/not-eval. For example, we can measure the discriminability of generated vs. production traffic, drive it down, and account for what remains. There’s early evidence this is doable: in Anthropic’s recent agentic-misalignment report, flipping single cues moved behavior dramatically, and eval-awareness showed up as a manipulable internal representation rather than a wall.
Finally, coverage-driven stress of the control stack (monitors, protocols, containment) doesn’t require fooling the model about eval status at all.
On pretraining: Yes, I think it is plausible that CDV will provide the broad base of data which will push you to the right “area” of the optimization space. And it will also let you audit that you used the “right” data. Finally, CDV helps in the “three-way investigation” (we arrived at acceptable risk / the V&V process is stuck / we are at strategic deception).
Expanding on the formation-window point, which is the part I am most interested in feedback on—here is the direction I’m considering (future post):
Model organisms of misaligned behavior are often constructed once and then studied. I’m considering running formation as a loop: Start with an RL environment containing a deliberately scripted oversight gap, so there is ground truth about what counts as exploitation. Then use coverage-driven patching to close the cheap reward hacks as needed, creating selection pressure toward exploitation that is more conditional on the oversight context and potentially more concealed.
Repeat this multiple times from the same start. On each pass, vary the oversight gap, the patching sequence and so on, and run the matched-pair checkpoint scan from my comment above. Then analyze the distribution of resulting changes against the coverage map of each pass.
Note that the main object under test would be the detection machinery, not the claim that we had made a schemer.
One possible failure mode is that the experiment trains increasingly sophisticated recognition of the artificial oversight cues, rather than reproducing anything relevant to real strategic concealment. I’m curious whether people think the direction is still useful, and what other failure modes they see.