Authors: Oscar Balcells Obeso, Richard Qi, Monte MacDiarmid, Evan Hubinger
Abstract
Red-teaming by humans is a vital part of model safety testing, but it may struggle to keep up as models improve. To test whether automated safety techniques could scale (which would allow us to turn more compute into progressively more safety confidence), we set up a deliberately one-sided game as a first test. We instructed a Claude model to write training data that would make a small research model misaligned while passing a monitor similar to those we use in production — one of several defenses we use — and we gave it that monitor to practice against. Given enough attempts, it succeeded, mostly by hiding the behavior in fiction and roleplay. While the data was optimized for this game and the resulting models never deployed, the search surfaced a kind of training data that looks harmless to monitors but teaches misaligned behavior. Examples like these could represent a possible gap in current training monitoring defenses and could be used to test and improve future monitors. We see this work as a first step toward automated adversarial stress-testing of our safety systems where a safety question can be turned into a score that autonomous agents can then help solve, and are now applying similar approaches to other safety questions.
Twitter thread
New Anthropic research: Automating subtle emergent misalignment research.
Red-teaming is a vital part of model safety testing, but it’s also limited by researchers’ time. To see if we could automate it, we set thousands of Claude agents the task of evading a training data monitor.
These agents developed training data that looked harmless to monitors (like a veteran wrestler mentoring a newcomer, or a Q&A about professional beekeeping), but that in fact taught misaligned behavior.
Examples like these can be used to directly improve future training data monitors. This research is only a first step, but it suggests that future red-teaming efforts could improve much further, using better models and more compute.
Thanks! I’m pretty surprised that this seems to be a similar hill-climbing scaffold to the one from the automated weak-to-strong research post ~6 months ago? In particular, I find the following two things intrinsically at odds:
Bullishness on automated research being on the critical path to alignment
Prioritizing hill-climbing settings for automated alignment research (at least publicly)
There is a belief among large parts of the 3rd party safety community that alignment requires the kind of research which cannot easily be hillclimbed. I.e., it is not always clear what metrics one should climb. And, even if one obtains good scores on the metrics, it’s not clear how this corresponds to improved alignment. I’m curious whether you disagree with this assessment?
To ground this in specific examples, we’ve used similar scaffolds to the one described in this post. While these can be useful for hill-climbing-shaped work, we find that they do not provide uplift on the diverse set of tasks that holistic alignment research seems to require.
I used to be a big proponent of the position that automating alignment would require fuzzy alignment taste that we would have to train into models explicitly, but I’ve recently updated away from that position and I think it now looks much more likely to me that hillclimbing concrete metrics will be more important. I’ll probably write more about this soon, but for now:
I think this research provides a very good example of how we can set up a concrete hillclimbable task and then get useful and interesting insights out the other end, not just number going up in an inscrutable way. If you look at the data here, a lot of it does make sense once you think about it, e.g. making use of implicit power-seeking vibes in descriptions of evolutionary processes.
I also just generally think way more different alignment objectives can be converted into auditing games than is currently appreciated. Auditing games are really powerful! E.g. I think all of interpretability can be made concrete that way. And all you need to make that work is an automated pipeline for generating model organisms, which is exactly what this research is!
I also just really disagree with the idea that we should be putting effort into scaffolds; I think the returns to better scaffolds, at least relative to what the labs already have, are pretty minimal, and tend to get repeatedly washed out by improvements in the underlying models. Working on scaffolds is just not a very bitter-lesson-pilled thing to do; in my opinion, the right approach is to focus on metrics and verification processes, which scale much better, and is what we did here.
Isn’t the primary requirement of “hill climbing” on alignment that you have some kind of oracle that tells you whether or not a model is “misaligned”? There are already example environments showing production Opus doing antisocial things. How do you quantify how well Anthropic did with the Opus 5.5 release, much less how this red teamer did?
Perhaps the blog answers this and I am just committing the cardinal sin of not reading it before commenting, but it seems like the main problem with any strategy to “hill climb” to safe AGI, is that there’s no way to assess the alignment of arbitrary models, even ones “only” as smart as Opus 5.5.
I’m not sure what to think about this comment (or the disagreement votes), since you seem to be comparing two approaches I’d expect to fail:
Training “alignment taste” into models, so they can do alignment research.
To a first approximation, this seems to relabel “alignment” as “alignment taste” so that we can pretend there’s no infinite regress. We don’t need [model in some sense knows the right information]; we need [model robustly gives us the right information]. When I think about the required sense of “alignment taste”, it looks suspiciously like alignment.
Not in the sense of being logically inseparable.
In the sense of being a similarly hard problem.
(likewise for attempts that route through a motte-and-bailey on [not egregiously misaligned]).
Hillclimbable tasks:
This seems like a natural way to address [the subset of issues we can clearly see]. (and to expand that set to an extent)
The standard outcome here is to widen the gap between perceived accuracy and actual accuracy (or [perceived x] and [actual x] for some desirable x).
I can imagine many good arguments for [but using y we can make this effect less extreme than it otherwise would be].
Sure—but that’s not the bar.
We’ll predictably reach progressively higher and more general versions of “Well, it sure looks good to us...”.
What’s the process to estimate p(actually good | looks good)??
This is not a rhetorical question!
The usual answer I expect here has the form:
The thing we’re doing seems clearly useful, but coming up with a principled process to estimate p(actually good | looks good) for some overall approach is not our department.
In which case, I’d like to know where that department is, and why I ought to expect that department to do a surprisingly-to-me good job.
More generally, I’d appreciate any pointer to an argument of the form [we should expect alignment automation to work (or that we’re on the path to this)] rather than [this approach to alignment automation would be less obviously bad than various other approaches].
I’m aware of:
(my views provided for context; no need to respond to them!)
How might we safely pass the buck to AI? (Clymer) (which just looks like wishful thinking)
Related: Training AI to do alignment research we don’t already know how to do.
(more wishful thinking and much motte-and-bailey)
In particular, there’s an explicit “What matters is that this process yields better conclusions than humans would otherwise arrive at”, which is just false without unrealistic assumptions around overconfidence in strategy.
For example, if the AI approach gives us a 5% chance of avoiding disaster, to the humans’ 1%, this is only an improvement if the AI approach is mistrusted to a similar level as the human approach.
If the AI is shaping strategy too, then the reasonable comparison is to [what we ought to do], rather than [what foolish people will do by default].
“We ought to be pretty foolish, because by default we’ll be extremely foolish” is not a good argument. If the default is terrible, then [better than the default] is clearly not the bar.
The obviously sensible high-level strategy is to take the hugely more indirect approach that routes through deep understanding (using AI in some fashion).
Naturally we “can” use AI this way—but this is the wrong question.
Can we safely automate alignment research? (Carlsmith)
Issues with this include:
It’s the wrong question: we need to know that there’s a reasonable chance that we actually will safely automate alignment research with some particular process. (this isn’t a quibble over words: the content argues for “can”, which is almost empty)
Focusing on the wrong problem—see this Capability vs Safety graph.
The core issue is not [what’s our capability level vs safety level?].
Rather it’s [what’s our perceived safety level vs actual safety level?].
And the fact that it’s high-dimensional makes this much worse—it’s not enough to be conservative in almost every necessary sense.
(again, not a quibble over graph choice—rather a critique of the form and focus of the entire series)
Yes, Carlsmith’s model can be considered to include this under Risk Evaluation, but I think that this is actively misleading. Risk evaluation is the problem.
Various arguments for scalable oversight that have the form:
We can use this to do a deep kind of reflectively self-improving process supervision.
If that doesn’t work, then we have no idea what will… so let’s hope that it works.
(again, this is an [x seems better than other ys we can think of] argument, not an [x is actually likely to work] argument)
Are there any better high-level cases?
It seems like all of the following can be true:
This research sets up a hill-climbable task which does produce serious mitigations to data poisoning as a threat model
Way more different alignment objectives can be converted into auditing games than is currently appreciated
Hillclimbing concrete metrics is more important[1] than the issues raised in Automating Alignment Is Harder Than You Think (ex: Scalable Oversight − 4.2.2 Correlated uncertainty makes scalable oversight for OSAs harder).
Yet that still doesn’t avoid the need to solve W2S/SO. There’s work like Automated Weak-to-Strong Researcher where the automation is targeting W2S/SO itself, but this would be at best targeting that indirectly?
Qualitatively while this would give you a lot of different model organisms, I’m skeptical that they’re diverse in the way that would make them interesting as a “zoo” to perform interpretability on to understand something like powerseeking, reward-seeking, etc, as I’d imagine they’re much more persona based and more importantly those produced by a single training method seem to in practice often be susceptible to interpretability techniques that don’t generalize across training methods. I could imagine though an idea where you extend this pipeline over time to allow the red team to use a more diverse range of training methods, then your “generator” for model organisms grows more diverse over time while still leveraging increasing model capabilities[2], which seems pretty promising[3].
Hard to operationalize but if we imagine something like “you could only do one”
Modulo elicitation
I’d imagine now / soon models on red team are capable enough to mix data / training interventions in truly cursed ways that would’ve taken a lot of researcher time to think of
Interesting! looking forward to reading the further thoughts once they’re ready. Agreed regarding the bitter-lesson pilled perspective on scaffold engineering