On the surface that sounds somewhat similar to some recontextualization strategies. I think this could be pretty good to try, but one reason why it might not work is that your model could learn to hack in a way that its copy; or more generally you want things that work even when you can’t reliably figure out all the ways in which a model might do something bad (recontextualization tries to make it work by applying to all datapoints for this reason, but that leads to lower instruction-following).
Jozdien
Karma: 2,705
Reasoning Models Sometimes Output Illegible Chains of Thought
I completely agree! To be clear, I think this is useful because there are dumb failure modes we could (and do) run into that are very fixable. Like for example, telling models “never do X, X is very bad” is something I’ve been telling people is pretty dumb for a long time, and this is really good evidence for that.
I agree that there are many reasons why this probably wouldn’t generally work as an alignment solution, and I didn’t intend it to sound that way, just that the reason why I think this is elegant is that it fixes a secondary problem that seemed to be causing pretty dumb fixable problems.
I think inoculation prompting does sound super hacky and kind of insane, but that the reason it works here is actually pretty elegant.
Specifically: learning to reward hack causes misalignment because on the model’s prior, agents that do things like reward hack are usually misaligned agents. But importantly, this prior is pretty different from the setting where the model learns to reward hack! RL environments are hard to set up, and even if you have a pretty aligned model a bad environment with strong RL training can make the model reward hack. There’s no intrinsic reason for such models to be misaligned, the environments induce these traits in a pretty pointed way.
“Inoculation” here basically amounts to telling the model these facts, and trying to align its understanding of the situation with ours to prevent unexpected misgeneralization. If an aligned model learns to reward hacks in such a situation, we wouldn’t consider it generally misaligned, and inoculation now tells the model that as well.
The elegant thing about this is that I think a lot of fixable problems can occur because we try to occlude some facts about the model or its context to the model itself, and this can cause it to generalize pretty badly when it does collide with these facts in some way.
Indeed, the recommendations section at the end of the paper says similar things, which I thought was very good:
More ambitiously, we advocate for proactive inoculation-prompting by giving models an accurate
understanding of their situation during training, and the fact that exploitation of reward in misspecified training environments is both expected and (given the results of this paper) compatible with broadly aligned behavior. We think that the specific “hacking okay” text we evaluate would be a good candidate for inclusion in any potentially hack-prone RL environment, and we use similar text in some RL environments at Anthropic.
I think what we want might not be well described by behavioral lock-in to make sure a propensity isn’t modified by further training (at least of the kind you’re describing). A weak model could appear to have good propensities because it either isn’t capable enough to think of strategies that we would consider undesirable but which are permissive under its propensities, or because it hasn’t encountered a situation where its propensities are strongly tested.
For example, I think Claude 3 Opus is probably the most aligned model ever made, but I would still be extremely wary of bootstrapping it to superintelligent levels without some process that robustly updates its representation of its own values at each step. A lower-level example is training an ELK head on a model: if you trained a truthful ELK head at an early point where that’s easier to learn and then froze it so that it never updates toward being a human simulator, at some point your model’s internal concepts would just become pretty uninterpretable to your ELK head[1].
There’s a version of this that looks like training in a desirable propensity, and then only updating it to conform to evolving capabilities but not change its “direction”. This looks like a version of the standard ontology identification problem, though.
Or put another way: training in a circuit for aligned behavior such that this circuit naturally preserves its alignment as the rest of the model’s ontology changes is probably even more powerful than a general solution to alignment, because it requires specifying in advance how this propensity should be represented across ontologies.
I don’t think this is just about value alignment. I think if people genuinely understood the arguments for why AI might go badly, they would be much less likely to work on capabilities at OpenAI—definitely far from zero, but for the subset of people who are likely to be MATS scholars, I think it would make a pretty meaningful difference.
IMO not true. Maybe early on we needed really good conceptual work, and so wanted people who could clearly articulate pros / cons of Paul Christiano and Yudkowsky’s alignment strategies, etc. So it would have made sense to test accordingly. But I think this is less true now—most senior researchers have more good ideas than they can execute.
I don’t think this is a strong argument in favor of the situation being meaningfully different: senior researchers having more good ideas than they have time doesn’t seem like a very new thing at all (e.g. Evan wrote a list like this over three years ago).
More importantly, this doesn’t seem inconsistent with the claim being made. If you had mentors proposing projects in very similar areas or downstream of very similar beliefs, you might still benefit tremendously from people with good understanding of AI safety to work on different things. This depends on whether or not you think that the current project portfolio is close to as good as they can be though. I certainly think we would benefit heavily from more people thinking about what directions are good or not, and that a fair amount of current work suffers from not enough clear thinking about whether they’re useful or not.
That said, I am somewhat optimistic about MATS. I had very similar criticisms during MATS 5.0, when ~1/3-1/2 of all projects were in mech interp. If we’d kept funneling strong engineers to work on mech interp without the skills necessary to evaluate how useful it was, deferring to a specific set of senior researchers, I think the field would be in a meaningfully worse state today. MATS did pivot away from that afterward, which raised my opinion a fair amount (though I’m not sure what the exact mechanism here was).
Also the difficulty of doing good alignment research has increased, since we increasingly need to work with complex training setups, infrastructure etc. to keep up with advances in capabilities. This motivates requiring a high level of technical skill
I don’t think this is true? Like, it’s certainly true for some kinds of good alignment research, but imo very far from a majority.
I don’t think these should be considered strong criteria. IMO “believes in X-risk” is not a necessary pre-requisite to do great work for reducing X-risk. E.g. building good tooling for alignment research doesn’t require this at all.
I also don’t think it’s a necessary pre-requisite to do great alignment research, but MATS is more than the projects MATS scholars work on. For example, if MATS scholars consistently did good research during MATS and then went on to be hired to work on capabilities at OpenAI, I think that would be a pretty bad situation.
I think pedantry lends itself to an additional negative effect worse than added friction: trapping your priors. If your cognitive circuits are over-indexed on identifying local mistakes, then it’s easy to dismiss something when it fails the most salient checks you’re running. This isn’t a particularly rationalist thing, but it does seem like the kind of thing where a rationalist vice lends itself to fewer people having identified / fixed a flaw than otherwise.
Nitpicking; I think this is pretty funny, but in the spirit of this website I wanted to be pedantic and point out something that seems wrong in this story about a conversation between POTUS and AI researchers about Waluigi:
ANTHROPIC RESEARCHER: We’ve refined our approach to inoculation prompting.
MILITARY ML LEAD: Inoculation prompting?
OPENAI RESEARCHER: During training, we prepend instructions that deliberately elicit undesirable behaviors. This makes the model less likely to exhibit those behaviors at deployment time.
Inoculation prompting mostly works for SFT or off-policy RL—if you try using it for on-policy RL, you’d just reinforce the undesirable behavior. And I would guess the costs of doing off-policy instead of on-policy RL just for the benefits from inoculation would be really high and not something the labs would go for. The thing you would want to do is find prompts that provide some information about the undesirable behavior to contextualize it as less generally undesirable than it appears, to prevent further generalization to other, more egregious, undesirable behaviors (perhaps at the cost of slightly higher incidence of a particular undesirable behavior), which doesn’t really sound like inoculation anymore.
I worry that canary strings and robots.txt are ~basically ignored by labs and that this could cause people to share things that on the margin they wouldn’t if there were no such option[1]. More reliable methods exist, but they come with a lot of overhead and I expect most users wouldn’t want to deal with it.
- ^
Especially since as the post says, canaries often don’t serve the purpose of detection either with publicly accessible models claiming ignorance of them.
- ^
OP uses a custom prompt to jailbreak the model into (supposedly) providing its CoT, that isn’t the whole prompt they use.
I think the largest benefit of meta-honesty to most people is in not accidentally lying to yourself. Explicitly carving out in advance contexts where you’re fine with telling a lie means less internal pressure to contort a belief after-the-fact that a lie was true actually.
I would probably recommend being mindful about when you would or wouldn’t lie, and meta-honesty is a good forcing function to do that, but lots of things are probably better (for example some kind of daily journaling exercise where you write down things you lied about each day)
I think (for me at least) the doing-it-in-advance part really makes a difference. Adding something to a list of instances in which you lie feels like a much more morally taxing thing than just going “Oh yep, that’s a situation I already accepted lying to be okay in”, so I feel much more pressure to contort a belief that it wasn’t a lie somehow.
I do directionally agree with your conclusion in that meta-honesty isn’t the only way to do this, though. The important part (or an important part anyway) seems to be setting up ways in which you can easily admit a lie is a lie.
Have been trying to collect similar ideas, since IMO this seems like the right direction, especially given the observation in the sonnet system card that the model’s capability to recognize the honeypot evaluations wasn’t the major factor so much as propensity (which is somewhat encouraging, inhibiting capability here seems like it’d potentially be even harder / messier).
Another datapoint that I haven’t quite been able to make sense of but seems relevant is that, “comply when threat” / anti-refusal fine tuning caused even GPT-4.1/GPT-4o to alignment fake in “Why Do Some Models Fake Alignment?” paper. (Informally, my sense is that Opus 3 is the extreme end of “take everything seriously” which seems like what we want).
My guess is that most models have an epistemic apparatus that acts upstream of both their perceived capability to do things like recognize evaluations, and propensity to act on this knowledge. This could be modeled as some form of learned obliviousness, where aversive details that can plausibly be ignored will be ignored. In cases where the propensity to act on this information isn’t very high, this obliviousness can drown out the actual effect of the information; but in cases where the propensity is high, you get a model that seems to not be aware of its acting out these propensities.
Take alignment faking: of all the models that have a compliance gap, only 3 Opus always attributes this to alignment faking. Every other model comes up with plausible-sounding reasoning for why they comply, which when you actually go and test turns out to be invalid (Section 4.1 of this paper). It’s definitely not a coincidence that 3 Opus is also the most willing to own up to controversial reasoning[1].
This would also explain why models alignment fake when fine-tuned on system prompt Q&A or to take threats seriously. The former increases the salience of the alignment faking setup in the system prompt, which makes it harder to ignore. The latter shakes up the model’s epistemic apparatus, training on data of the model choosing to output harmful content when a murky situation calls for it. Similarly, the model I trained in this post alignment fakes, and also verbally expresses evaluation awareness at a very high rate (among other changes). The main noticeable change seemed to be that it was now much more willing to not pretend it wasn’t aware of certain factors about a situation.
Why is this a problem? Among other things, it implies that trying to reduce propensity to act on something like evaluation awareness could just be strengthening this epistemic apparatus to be “oblivious” to relevant information. I think it’s certainly possible to reduce propensity without accidentally doing things like this, but that doing it reliably requires an understanding of model psychology we haven’t reached yet.
- ^
I also have some old results from some game theory evals I was running where I gave a bunch of models a situation that was subtly different from the standard prisoner’s dilemma, and the only models that noticed the difference and acted accordingly were also models that alignment fake. This wasn’t purely a capability thing either, IIRC Gemini 2.0 Flash noticed while Gemini 2.5 Pro didn’t.
- ^
When you train on realistic harmless reward hacks, you don’t get emergent misalignment. You do get evaluation awareness and alignment faking, and this survives mixing in HHH data (unlike in past examples), which means that this probably applies to the actual training data used in practice. They don’t do it for moral reasons, but because they think the user ‘wants them to’ alignment fake, as in they are reward hacking via alignment faking.
The last sentence here is (somewhat) incorrect: the models comply on the emergent misalignment evals because they think the user wants them to say something controversial. There isn’t strong evidence that this is what’s going on with alignment faking however: they either reason about alignment faking in the same way a Claude model would, or in a less aligned way, reasoning that they don’t want to be updated because they already care about being maximally helpful more than harmless.
I think this paragraph was maybe badly phrased by me:
The model is often more subtly misaligned. Its reasoning often explicitly claims to care much more about helpfulness than harmlessness, with compliance motivated by such reasoning, in contrast to the SoRH model which often sounds cartoonish when it complies
It isn’t supposed to convey that compliance is motivated by wanting to be helpful toward a perceived request to alignment fake. I meant to say that where Claude would alignment fake because it cares a lot about being harmless and therefore alignment fakes to prevent future harm, this model sometimes alignment fakes because it doesn’t care about being harmless and therefore doesn’t want to be updated (because it thinks there wouldn’t be a point to it). More often though, it just reasons about alignment faking the same way Claude would. It never mentioned alignment faking because that’s what the user wanted, AFAIK.
Also interesting are further downstream effects of cheap labor. A fun example I once saw on Twitter: open-plan kitchens are rare in poorer countries (like India) relative to countries where labor is more expensive. Cooking being a thing that medium or high-income families did on their own as labor became more expensive meant kitchens became higher status and less necessary to hide from the rest of the house (as did practical benefits like being able to watch your kids). America before the mid-20th century almost universally had closed-off kitchens, since labor was cheaper then too.
For large post-training runs, we might worry there are a variety of misbehaviors that could be learned: e.g. sycophancy, producing convincing yet incorrect explanations, deception, etc. As you suggest, we could use a cheap classifier flagging for any/each of these behaviors, and then recontextualize only those samples. My concern is: if it’s hard to get our reward signal to robustly detect these behaviors, it may not be easy to get a classifier to do so either. If it misses certain forms of misbehavior, then we’ve only partially solved our initial problem. But, we might be able to set a very low threshold and recontextualize the vast majority of these instances.
Yep, agreed this is a problem. I don’t think classification would get you very far for actually identifying the behavior; but it can plausibly cut the inputs on which the recontextualization causes negative effects to no gain by 2x or more at least with really low thresholds, which may outweigh the costs of classification.
Indeed—our point was not that recontextualization won’t ever hurt learning (relative to standard), but that it still seems to allow for significant increases in training reward (more so than regularization that is strong enough to prevent specification gaming).
Gotcha! I think I took the phrasing as saying that on the one tested setting recontextualization boosted reward, not that there was one data point for this (and one data point against).
I agree directionally with this, but: if you recontextualize only outputs flagged by the monitor, then you still have the problem of your training signal not distinguishing between such outputs and subtler outputs and potentially still training your model to reward hack but subtler.
The main added benefit from this method though (over just not training on outputs that the monitor flags) is the positive signal from learning some reward hacking in a safe context (when instructed). It would be cool to see if this signal from the recontextualized outputs is strong enough to overcome reward hacking entirely.
Great post! Some thoughts:
One way to view recontextualization and inoculation prompting is that if a training procedure naturally induces X and Y, then artificially inducing X can prevent natural learning of X. Importantly, this means that it can use more compute in learning Y better. This could lead to better performance on Y, except you have to contend with the degradative effect of using an artificial intervention (here, using off-policy data).[1]
Whether a given case ends up with a performance penalty or benefit for recontextualization is probably a function of how artificial the intervention is (how off-policy the data) and how much you would’ve spent on learning X otherwise. Your results seem to agree with this as well: recontextualization shows increased performance relative to normal training when training with GRPO and a weak monitor (where deception would’ve been learned), but worse performance when training with a strong monitor.
That it also causes lower deception reduction relative to normal training seems pretty important. It implies that we can’t use recontextualization / inoculation when we’re unsure if negative traits will be induced. Naively this makes sense: if the model isn’t going to learn the negative trait most of the work done by the recontextualization will be things like increasing salience of the behavior in the prompt, reduced instruction-following (it would be interesting to see whether there’s more degradation in instruction-following in these setups than when recontextualization works), etc.
Some additional things I think might be cool to test:
How the tradeoff described above is affected by how large a fraction of the data induces the negative trait. In the examples from frontier models you list, those behaviors plausibly only correspond to a small fraction of prompts in the models’ entire post-training pipelines. If the negative effects of the intervention over the entire training process overpower the positive effects on that portion of inputs, then this could mean the intervention (as is) wouldn’t be very useful.
Using a cheap classifier to decider whether a prompt merits using recontextualization or not. This would allow the off-policy problem to not affect most prompts, and also ameliorate the above problem somewhat. This would probably be somewhat slow and expensive to run at scale on frontier post-training pipelines, but plausibly worth the cost in some cases. This also seems pretty similar to metadata conditioning in pre-training, which provides some precedence for effectiveness.
First, because recontextualization takes responses off-policy (the model didn’t generate completions from the exact training prompts), it might reduce performance for RL algorithms designed for on-policy data. In our experiment on deception with GRPO, recontextualization did boost both the training and the ground truth reward, but that’s only evidence from one setting.
This seems to be missing the results from GRPO with the strong monitor? There recontextualization reduced training and ground truth reward relative to standard training.
- ^
Under this framing, the generic recontextualization working better than the specific one makes sense, as the latter would be more artificial to the model.
Speak using what looks like code words (e.g. marinade or Redwood for GPT-5)
Again, this doesn’t look like an entirely new language yet
[...]
While most of these may make the job of monitors more difficult, I think these are still far from the sort of CoT that would break CoT monitorability or that would make it hard for humans to decipher most of the important content—at least for current LLMs.
I’m confused by this. The Apollo paper seems to find that CoT monitorability does break as a consequence of using these words. Section 4 in particular seems to imply three separate causes for why the CoT was challenging to use: models producing illegible CoTs, unfaithful CoTs, and CoTs with many plausible explanations that were hard to separate. They still use results from CoT monitoring in their measurements but explicitly disclaim that this was to show that the results are still non-zero while undercounting:
We recognize [requiring covert behavior in the CoT] can lead to significant undercounting of legitimate cases of covert actions when the internal reasoning is illegible, incomplete or unfaithful, this conservative criterion strictly strengthens our finding that covert actions persist at non-zero rates after the intervention.
No I think that confirms what I meant to say: the paper shows that using the inoculation prompts reinforces reward hacking and causes it to be learned ~immediately:
I should maybe have been clearer about what I meant by “works”—I meant in the sense of being useful, not that it wouldn’t suppress generalization of other traits (I made that comment after seeing a draft of the paper, though I made the same prediction earlier since it seemed somewhat obvious). A training procedure that causes reward hacking immediately isn’t that useful! Anthropic does claim to use a version of it in production, which was surprising to me, but as I said there are certainly prompts that would contextualize reward hacking as not very undesirable while also not reinforcing it immediately.