I also found this encouraging (especially his emphasis on empirically testing whether alignment generalizes and survives further capabilities training). Coming from complex-system verification, I think coverage-driven methods could help improve (and evaluate) generalization: Use an explicit, evolving map of situations and risks to guide evaluation and alignment training, and refine the map upon failures and newly-discovered risks. I explored this in an earlier post on coverage-driven alignment, building on “Teaching Claude Why.” I’d be interested to know how closely it resembles labs’ internal practices, and what people think of this direction in general.
Yoav Hollander
Karma: 90
Thanks Josh. Answering in reverse order:
On 3: I’d actually push back on your update (though unsure, and the post-mortem may prove me wrong). Those were internal experimental RL models, mid-training, guardrails off – and as you note, we don’t know whether they had the relevant safety post-training at all. If the broad alignment base was absent, this incident is ~zero evidence against your “more robustly aligned region” claim: It shows what happens without the base, under exactly the RL pressure where durability is the open question. So your original comment may survive intact. BTW, the fact that neither of us can tell which configuration this was is itself an instance of “verify all relevant configurations” – we’re missing the instrument, not just the data.
On 2: Agreed, and I’d sharpen it: These are not just two important areas but two dimensions of the coverage map (task-satisfiability × guardrail-configuration), so their combinations get exercised systematically (e.g. “impossible task while guardrails are off” is exactly the bucket where this incident lived).
On 1: Mostly agree (but here too I am waiting for the post-mortem). Some of this would plausibly have prevented the initiating conditions (spec-bug search catches the impossible tasks, the escalation channel gives stuck agents a legal move). But for the message board itself I’d only claim detection – coverage-driven checking should have surfaced it in days rather than months, and rewind semantics bound the damage. “Detect early and bound the blast radius” is the defensible version.
V&V takes on “Pacing the frontier”
Thanks – I am tempted to continue the series with shorter, focused posts – e.g. this one.
V&V takes on OpenAI’s long-horizon incidents
Expanding on the formation-window point, which is the part I am most interested in feedback on—here is the direction I’m considering (future post):
Model organisms of misaligned behavior are often constructed once and then studied. I’m considering running formation as a loop: Start with an RL environment containing a deliberately scripted oversight gap, so there is ground truth about what counts as exploitation. Then use coverage-driven patching to close the cheap reward hacks as needed, creating selection pressure toward exploitation that is more conditional on the oversight context and potentially more concealed.
Repeat this multiple times from the same start. On each pass, vary the oversight gap, the patching sequence and so on, and run the matched-pair checkpoint scan from my comment above. Then analyze the distribution of resulting changes against the coverage map of each pass.
Note that the main object under test would be the detection machinery, not the claim that we had made a schemer.
One possible failure mode is that the experiment trains increasingly sophisticated recognition of the artificial oversight cues, rather than reproducing anything relevant to real strategic concealment. I’m curious whether people think the direction is still useful, and what other failure modes they see.
Thanks. I agree that against a fully formed adversary which can recognize and adapt to measurement, behavioral coverage of a fixed system is not enough. Which is why control is so important.
I’d push back on “the only meaningful coverage axis is eval/not-eval” in two ways:
First, eval/not-eval may be an axis across formation, not just across situations. One reasonable formation story is that strategic deception is a learned combination of (tracking the oversight regime × conditional behavior × suppression of revealing behavior), and under gradient descent it’s hard for that to arrive fully formed. So intermediate versions may leak, and scanning back across the checkpoint series with matched scenario pairs that differ only in oversight cues can turn eval/not-eval into a probe axis during that formation window. Note that this only works if strategic deception arrives incrementally via SGD rather than all-at-once via in-context reasoning, and if early versions can’t already recognize and game the probe itself.
Also, “eval” isn’t one Boolean—it is a bundle of realism cues, and coverage over the cue space is measurable even when you can’t fully hide eval/not-eval. For example, we can measure the discriminability of generated vs. production traffic, drive it down, and account for what remains. There’s early evidence this is doable: in Anthropic’s recent agentic-misalignment report, flipping single cues moved behavior dramatically, and eval-awareness showed up as a manipulable internal representation rather than a wall.
Finally, coverage-driven stress of the control stack (monitors, protocols, containment) doesn’t require fooling the model about eval status at all.
On pretraining: Yes, I think it is plausible that CDV will provide the broad base of data which will push you to the right “area” of the optimization space. And it will also let you audit that you used the “right” data. Finally, CDV helps in the “three-way investigation” (we arrived at acceptable risk / the V&V process is stuck / we are at strategic deception).
When is misalignment just a bug?
I come from an adjacent field—verification and validation of “physical AI”—so I read this partly as a verification problem. On genuinely unsupervisable fuzzy tasks, such as judging novel research agendas or paradigm-level reframings, I don’t think V&V gives a magic answer—those seem as hard as you say.
But I think one important subproblem in your bucket—assessing whether a system behaves acceptably across the situations that matter—is more supervisable than the framing may suggest. Coverage-driven verification lets you turn “did we exercise the relevant situations, and did behavior hold across them?” into a measured property rather than a single global judgement. Mature V&V also has machinery for validating the checker itself, and an active (human + AI) process for discovering missing coverage dimensions—the “spec bugs” that are obvious in hindsight but not enumerable upfront (see link below).
This doesn’t touch correlated-evidence aggregation. It helps only partially with paradigm choice (it can surface new coverage dimensions, though not replace the framing they sit in). On proxies it helps with coverage, not with whether the proxy is relevant to alignment at all. And coverage discovery is unbounded—not a proof of completeness. But assuming scheming is off the table (as you suggest for this discussion), many concrete failures, once found using the machinery above, convert into ordinary spec checks. So I’d suggest the genuinely irreducible part is narrower than the full fuzzy-task set, concentrated where a system’s behavior changes because it’s being measured.
Fuller version here.
Thanks Yuxin—and thanks especially for the description in your post of the mapping into “Robot SOTIF”, and how that might play in China’s standards-driven environment. You also wrote that if the coverage maps and risk assessments produced by CDV can become evidence of “reasonable care”—just like safety cases in autonomous driving—then alignment V&V gains an institutional incentive base.
That incentives part is outside my area of expertise, and yet it is crucial. Without it, rigorous alignment V&V loses to “ship faster” (in all jurisdictions). In AV-land what makes expensive, systematic V&V rational is the well-established “incident → investigation → someone is liable” loop, but there’s no AI analog yet.
The most promising hook I know of is the work trying to close the “responsibility gap” by attaching an AI agent’s actions to a human or corporate principal. Note this is mostly not aimed at the labs building general-purpose models, but rather at whoever deploys a specific AI-for-something (an AI CEO, a medical AI, a delivery robot) and thereby becomes the identifiable principal (and that specificity also makes the coverage map more tractable). If that holds, CDV-style coverage maps and risk-per-bucket claims can become exactly the “reasonable care” record such a principal would need.
If anyone reading this works on AI governance, algorithm assessment, or liability and sees a way to make rigorous V&V the path of least resistance rather than a cost center, I’d very much like to talk.
Your adversary—“agentic RL with misspecified rewards”—is exactly what I’ve been working on, from a different field (coverage-driven verification in AV and chip safety). One distinction that might be useful: Misspecified rewards split into ones you could have anticipated (findable by denser testing) and ones where the spec was simply silent on a contingency (price collusion, a mid-campaign rule change, a move nobody enumerated). The second kind is more dangerous, because a stress-test built from misspecifications you can author can’t contain them by construction. The post I just wrote about it calls the second kind “spec bugs”, talks about “can you enumerate the dimensions of an open agent”, and suggests enhancing “Teaching Claude Why” with a coverage-driven adversarial RL pipeline. Here it is—would be curious what you make of it.
Thanks David. I’d put it slightly differently: CDV isn’t trying to make the model more ethical through iteration (you’re right that experience doesn’t make a CEO more ethical). It is trying to find out where the character training actually held and where it silently didn’t. Even if you’re fully betting on character, you still need to know whether that character generalizes to (say) the low-oversight / conflicting-incentive / multi-agent region, and the only efficient way I know to find that out is via systematic, CDV-style sampling.
I’d actually argue virtue ethics is the case where this matters most, not least. A character bet leaves the spec maximally implicit: “be of good character” says nothing explicit about price collusion or a mid-campaign rule change. Those are what I call spec bugs in the post—regions nobody thought to enumerate. So the more you lean on character rather than an enumerated behavioral spec, the more unmapped territory you have, and the more you need coverage discovery to surface it.
Finally, you are absolutely right that “guarding against alignment deterioration during RL” is another important consideration. In fact, while writing the post I challenged myself to come up with techniques which have a chance of countering @evhub’s long-horizon RL fears (e.g. the AI CEO).
Here is a new post I wrote specifically about handling the hard AI-CEO case. Comments welcome.
Coverage-driven alignment—What ‘Teaching Claude Why’ can borrow from AV verification
Right – I was also associating this in my mind with his ‘generalization science’ suggestion. However, I think he mainly talks about measuring / predicting generalization (and so does the referenced “Influence functions” post). My main thrust (see also the paper) is a principled methodology for causing the model to “generalize in the direction we want”.
V&V from physical autonomy might address a meaningful slice of the long-horizon RL / deceptive-alignment worry. In AVs we don’t assume the system will “keep speed but drop safety” at deployment just because it can detect deployment; rather, training-time V&V repeatedly catches and penalizes those proto-strategies, so they’re less likely to become stable learned objectives.
Applied to AI CEOs: the usual framing (“it’ll keep profit-seeking but drop ethics in deployment”) implicitly assumes a power-seeking mesa-objective M emerges for which profit is instrumental and ethics is constraining. If strong training-time V&V consistently rewards profit+ethics together and catches early “cheat within legal bounds” behavior, a misaligned M is less likely to form in the first place. This is about shaping, not deploy-time deterrence; I’m not relying on the model being unable to tell test vs deploy.
A plausible architecture is A builds B, where a more capable model A is trained via repeated “build a system + verify it” cycles (meta-V&V), shaping A to value the verification process itself. A then constructs narrower agents B (e.g., CEO). When A > B, verifier/verified asymmetry plus direct V&V on B gives real leverage, provided audits are hard to anticipate, and specs/coverage are good.
AI CEOs (and similar long-horizon agents) are obviously much harder to spec, simulate and check than AVs. I assume both AIs and people are involved in the iterative V&V / spec refinement loop—see more background on V&V for alignment here.
Not claiming a solution: V&V can’t guarantee inner alignment; specs/coverage are hard; defense-in-depth is needed; and A > B may fail near AGI. But in this long-horizon RL setting, training-time V&V could (60% confidence) substantially reduce inner-misalignment risk by ensuring power-seeking/deception get caught early, before they become stable learned objectives.
Got several variants of the following question: “You say you have no solution for scheming, which could make the V&V method unworkable. Why do you then claim it is a practical method?”
Here is my answer: Scheming could in theory break any oversight method, but the V&V method does make several anti-scheming contributions:
(1) Its scenario-based, coverage-driven tests flush out most specification gaming tricks (a milder form of reward hacking) automatically, essentially treating them as bugs (see § 4.2) (2) It plugs into other frameworks—e.g. in CAI it supplies the “external reality check” that prevents model-collapse (see § 3.1). (3) Because every run produces a human-auditable safety-case and logged sim traces, and because it is easy to request virtual sims of new scenarios, spot-audits are cheap—giving schemers a constant risk of exposure. (4) These spot-audits can make various anti-collusion approaches (like whistle-blower, rotation and diversity) more effective.
That is why I still call it a practical layer, even while we attack scheming head-on elsewhere. Would love to hear opinions about these points.
The V&V method—A step towards safer AGI
Summary: My intuition is that “High Reliability Organizations” may not be the best parallel here: A better one is probably “organizations developing new high-tech systems where the cost of failure is extremely high”. Examples are organizations involved in chip design and AV (Autonomous Vehicle) design.
I’ll explain below why I think they are a better parallel, and what we can learn from them. But first:
Some background notes:
I have spent many years working in those industries, and in fact participated in inventing some of the related verification / validation / safety techniques (“V&V techniques” for short).
Chip design and AV design are different. Also, AV design (and the related V&V techniques) are still work-in-progress – I’ll present a slightly-idealized version of it.
I am not sure that “careful bootstrapped alignment”, as described, will work, for the various reasons Eliezer and others are worried about: We may not have enough time, and enough world-wide coordination. However, for the purpose of this thread, I’ll ignore that, and do my best to (hopefully) help improve it.
Why this is a better parallel: Organizations which develop new chips / AVs / etc. have a process (and related culture) of “creating something new, in stages, while being very careful to avoid bugs”. The cost-of-failure is huge: A chip design project / company could die if too many bugs are “left in” (though safety is usually not a major concern). Similarly, an AV project could die if too many bugs (mostly safety-related) cause too many visible failures (e.g. accidents).
And when such a project fails, a few billion dollars could go up in smoke. So a very high-level team (including the CEO) needs to review the V&V evidence and decide whether to deploy / wait / deploy-reduced-version.
How they do it: Because the stakes are so high, these organizations are often split into a design team, and an (often bigger) V&V team. The V&V team is typically more inventive and enterprising (and less prone to Goodharting and “V&V theatre”) than the corresponding teams in “High Reliability Organizations” (HROs).
Note that I am not implying that people in HROs are very prone to those things – it is all a matter of degree: The V&V teams I describe are simply incentivized to find as many “important” bugs as possible per day (given finite compute resources). And they work on a short (several years), very intense schedule.
They employ techniques like a (constantly-updated) verification plan and safety case. They also work in stages: Your initial AV may be deployed only in specific areas / weathers / time-of-day and so on. As you gain experience, you “enlarge” the verification plan / safety case, and start testing accordingly (mostly virtually). Only when you feel comfortable with that do you actually “open up” the area / weather / number-of-vehicles / etc. envelope.
Will be happy to talk more about this.
I like the embedded-evaluator proposal. Coming from chip and autonomous-vehicle verification, I’d suggest one concrete addition: An evolving coverage map connecting safety claims to evidence.
For each relevant configuration (during training, internal use, and release), record which requirements and situations were tested, what failed, what remains unchecked, and where the checking methods themselves are weak. Also record whether apparent alignment survives further capabilities training and generalizes to situations withheld from alignment training.
The map may be used to guide improving alignment, not just measuring it—for example, by systematically generating alignment stories / training cases across relevant situations. When a problem is found, identify the broader failure class, strengthen alignment across that class, and re-evaluate (including after further capabilities training).
A coverage map cannot establish that all important risks have been identified: Searching for missing dimensions and checkers is part of the work. But it can make the scope and limitations of the evidence inspectable, and help prioritize how to use the time pacing buys us. I discuss this approach in V&V takes on “Pacing the frontier”.