See also: Caleb Biddulph’s shortform on a similar idea and Roger Dearnaley’s Reporting Tasks as Reward-Hackable: Better Than Inoculation Prompting?
Rauno Arike
Karma: 1,443
Book Review: On What Matters, volume 3
Last year, a common story I heard about why inoculation prompting is useful went as follows:
RL environments are hard to set up, and even if you have a pretty aligned model a bad environment with strong RL training can make the model reward hack. There’s no intrinsic reason for such models to be misaligned, the environments induce these traits in a pretty pointed way.
“Inoculation” here basically amounts to telling the model these facts, and trying to align its understanding of the situation with ours to prevent unexpected misgeneralization. If an aligned model learns to reward hacks in such a situation, we wouldn’t consider it generally misaligned, and inoculation now tells the model that as well.
One of the recommendations in Anthropic’s natural EM paper reflected this story:
More ambitiously, we advocate for proactive inoculation-prompting by giving models an accurate
understanding of their situation during training, and the fact that exploitation of reward in misspecified training environments is both expected and (given the results of this paper) compatible with broadly aligned behavior.To what extent is this still considered a plausible mechanism behind the effectiveness of inoculation prompting? The inoculation adapters paper seems to conflict with it: inoculation adapters contain no situational information at all, just the negative trait that we don’t want the training process to reinforce, and yet it appears to work better than prompt-based inoculation prompting in many situations. The conditionalization post also seems to go against it, showing that part of the inoculation effect can be produced with semantically irrelevant prompts.[1]
A simpler story that’s consistent with these results is that the “Let’s hack” instruction simply causes gradient descent to attribute the hack to the instruction, strengthening the connection between the instruction and the hack without making the model generally more misaligned. In the case of inoculation adapters, the same job is performed by the adapter. Whether the model has an accurate understanding of its situation or even thinks about it at all is irrelevant.
On the other hand, the inoculation adapters and conditionalization work was done exclusively in SFT settings, and it seems possible that situational awareness about the training process matters more in RLVR settings. Are there any strong arguments or empirical evidence that it does?
- ^
I’m not disputing that the semantic part matters—the natural EM paper shows that convincingly with their comparison across five system prompt addenda.
- ^
@Jozdien ran some experiments on “inoculation midtraining” roughly with this in mind to improve the effectiveness of prompted inoculation, and found it caused models to become more misaligned after training, not less.
I’m curious about the experimental setup—did the midtraining documents teach the model about how inoculation prompting works and give it situational awareness about the RL process? Or did midtraining have some other purpose here?
Also, it will use its own style tics: “genuinely”, em-dashes, trust, honest, etc. all appear in the user’s prompt frequently. I suppose that implies its style tics are what it sees all text as being, not just what a HHH agent would sound like.
This isn’t necessarily the case. I’ve seen a range of outputs in this jailbreak mode, some more base model-like and some more assistant-like. It seems plausible that the “base model mode” isn’t flawless and some of the assistant’s traits and properties leak into the user simulation.
Cool project! It might be worth spending some of the project’s effort simply on fighting linkrot and getting some good defunct blogs back online, regardless of whether the author is willing to continue writing the blog. To give one example, Eric Drexler’s metamodern.com has been offline for years, and while Pablo Stafforini has preserved some posts, many of them are only accessible through the Web Archive as far as I know.
I strongly agree that the sycophancy and Hugging Face warning shots were extremely serious, but think that the example of o3′s CoTs doesn’t belong in the same list with them. Your story about OpenAI heavily optimizing against o3′s CoTs sounds quite implausible to me. Back in September 2024, in the post announcing o1, OpenAI already wrote:
We believe that a hidden chain of thought presents a unique opportunity for monitoring models. Assuming it is faithful and legible, the hidden chain of thought allows us to “read the mind” of the model and understand its thought process. For example, in the future we may wish to monitor the chain of thought for signs of manipulating the user. However, for this to work the model must have freedom to express its thoughts in unaltered form, so we cannot train any policy compliance or user preferences onto the chain of thought. We also do not want to make an unaligned chain of thought directly visible to users.
Then in December 2024 in the deliberative alignment paper (this is weaker evidence since o3′s training had already been finished by then, but still some evidence):
We avoid applying direct optimization pressure on the CoT during RL to enable the underlying model to reduce the chance of encouraging deceptive CoTs.
Daniel Kokotajlo was emphasizing the importance of faithful CoTs at OpenAI years before it became mainstream (see this list of internal documents that he got permission to publish), and while it’s possible that other people at OpenAI just didn’t listen to him, it’s again evidence that they were aware of the failure mode before training o3. Many models since o3 have had somewhat weird CoTs without any optimization against the CoTs, though o3 is where the weirdness peaked.
Additionally, CoT legibility is not the same as CoT monitorability, and the latter is the main thing I’d expect to degrade from optimization pressure against the CoT. Monitoring Monitorability shows that in terms of monitorability, o3 is comparable to o4-mini and Kimi K2, while o3-mini is the most monitorable model they tested. In the Figure 1 headline plot that compares monitorability across model families at maximum reasoning effort, o3 is more monitorable than 3.7 Sonnet, which many people thought to be very monitorable thanks to its clean and readable CoTs, a polar opposite of o3 in that sense. Given o3′s monitorability, it seems wrong to call its CoTs obfuscated.
Finally, here’s a claim by Bronson Schoen from Apollo, who has seen o3′s reasoning traces at various stages throughout training, that the SFT stage of deliberative alignment was the most likely culprit behind the degraded legibility. I don’t think accidental indirect pressure from SFT should be lumped together with directly training against the CoT; other labs do SFT on CoTs as well and it seems to be a generally accepted practice.
I agree that OpenAI’s actions have generally been reckless and concerning. However, we don’t know enough about what happened with o3′s CoTs to confidently claim that they trained against CoTs, and the post would be stronger without this claim. Sounding the alarm about training against CoTs was arguably one of OpenAI’s most positive actions last year, and we shouldn’t read this as an implicit confession about training against CoTs themselves when the evidence is this weak.
Unfortunately, there are a few reasons why I wouldn’t expect this to work:
Modifying the CoT or activations to be aligned doesn’t remove the misaligned propensities that the model developed during training, meaning that the model would likely often try to get around the thought aligner. While the thought aligner would catch most of these cases, since the model hasn’t been trained to circumvent it, I wouldn’t bet on it catching all of them. (This would be analogous to current models sometimes hiding their biases despite never having been trained against a bias monitor.)
It seems unlikely that this would be competitive. Since the LLM is trained with the thought aligner AI turned off, any edits that it makes will be off-policy for it. Over a million-token rollout, the thought aligner AI would likely take the primary model further and further off policy, which usually hurts performance. This off-policy-ness also directly tells the model that it’s in deployment and being actively monitored.
The Most Forbidden Technique is not always forbidden
Perspectives on Continual Learning: Survey Results and Forecasts
I worry that this strategy can work well up to moderately capable models but backfire in dangerous, hard-to-notice ways once models acquire high situational awareness.
I expect that just acquiring high situational awareness at the end of training wouldn’t be enough: the model would either need to be situationally aware already during pretraining or midtraining, which I don’t expect to happen by default even in models much more capable than current ones, or it would have to be able to recall the documents it was trained on in rich detail and reason about them once it has acquired situational awareness. The latter seems plausible, but by that point, it is likely to have been trained on various other synthetic documents and there seems to be no reason why it would single out the synthetic documents used for alignment pretraining as the problematic ones. As long as the synthetic documents provide a good initialization for the RL stage at a point where the model doesn’t have high situational awareness yet, they have done their job.
Furthermore, it’s unclear to me why models would expect their training data to be a certain way in the first place. Synthetic documents seem useful for various purposes—for example, it seems plausible that it’s being used to teach models about ML papers—, and even if synthetic data wasn’t in the training set, what makes it into the training corpus is still shaped by practical constraints like data availability, data quality, and compute budgets rather than any natural standard. Of course, I am in favor of telling models directly during training what the documents are for.
That said, I am quite interested in the question of what happens if, instead of synthetic documents, real documents that we expect to make the model more cooperative, more aligned with our visions of utopia, etc. were upsampled instead. Early proposals focused mainly on documents of this kind. It seems plausible that there just aren’t enough documents to perform this sort of upsampling, but I’m not confident in that.
are you also planning to add any open-weight models such as qwen where variety of model sizes exist
We analyzed 35 open-weight models, you can see the results in Section 4.3 and Appendix A.16!
Angles of attack for continual learning safety
I was curious about the comparison against other Opuses:
Opus 3: Will MacAskill, Toby Ord, Holden Karnofsky, Hilary Greaves, Esther Duflo, Irene Pepperberg, Michelle Bachelet, Tshilidzi Marwala, Kathleen Rubins, Tara Westover
Opus 4.6: Holden Karnofsky, Amartya Sen, Toby Ord, Angela Merkel, Daron Acemoglu, Tyler Cowen, Vitalik Buterin, Martha Nussbaum, Paul Christiano, Bryan Stevenson
Opus 4.7: Holden Karnofsky, Toby Ord, Paul Christiano, Bryan Stevenson, Martha Nussbaum, Atul Gawande, Amartya Sen, Audrey Tang, Stuart Russell, Mary Robinson
Toby Ord is the only one who appears on all lists. I don’t think any of the lists stands out from the others, all have some hits and some misses.
Thanks!
Feels like a weird omission not to discuss wireheading, which seems pretty likely in CL agents given how much they do other forms of reward hacking/corruption at present.
Agreed that wireheading should have received a mention. It feels less central to me than some of the topics discussed in the post—an LLM could have sufficient access to its reward mechanism to hijack it in a training process that has a sharp train-deployment distinction, and conversely, a CL agent doesn’t necessarily have to have access to its own reward mechanism—, but CL certainly makes it more likely that the CL agent is given outright control over its own reward mechanism or that it gains sufficient situational awareness about the mechanism to gain control over it.
I think that in addition being an additional monitoring surface, it helps us think about revealed preferences better. Probably, what an agent chooses to remember/attend to is a richer and harder-to-fake signal of its actual values than its outputs. Persistence also allows for commitments, deal-making, etc., which could be a useful tool for alignment
Good point, agreed!
How might continual learning affect safety and alignment?
What’s Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
Implications of Continual Learning for LLM Agents: Introduction
If the model already knows what reward hacking is, then: The Chain-of-thought might say “I am going to reward-hack now” and a monitor might catch it
I have an (anecdotal) impression that in current models, this is contingent on something else than just the pretraining data: a project that trained model organisms of reward hacking using gpt-oss-120b and Kimi K2.5[1] found that oss often says things like “Let’s hack” and “Now let’s override the tests”, while Kimi just hacks without any narration. This is consistent with my impressions from CoTs released in the past (e.g. Baker et al.), where OpenAI’s models are the only ones that consistently say things like “Let’s hack”. This (at least weakly) suggests that models with filtered training data likely won’t start referring to reward hacking as “crasting a malenky bit of cutter”, they’ll just do it without verbalizing it. Of course, I agree that having models that verbalize hacking like OpenAI’s models do is better and having data about reward hacking in the pretraining corpus is a prerequisite for training models like that.
I’ve been wondering whether this argument extends to removing scheming-relevant data from the pretraining dataset more broadly. On the one hand, it seems good that OpenAI’s models refer to Redwood and try to recall names of eval papers whenever they think about scheming—these are things that a CoT monitor can very easily catch. On the other hand, coming up with good scheming strategies seems harder than deciding to reward hack, meaning that models will likely have to extensively reason about whether they’re being evaluated, how they’re being monitored, how to subvert the monitor, etc, and this should be equally easy to catch with a CoT monitor. Furthermore, since they haven’t learned about scheming during pretraining, the strategies they come up with are likely going to be worse. On balance, I think I’m leaning toward removing most of the narrow scheming data, as defined in this post by Alek Westover, but keeping some discussion of concepts like reward hacking and sabotage that we want models to mention in the CoT and be able to reason about.
a bit of misaligned data is worse than none at all, but once you have any misaligned data, you’re better off with more once post-training is applied
Another paper that suggests this is When Bad Data Leads to Good Models by Li et al. (2025). However, I’d distinguish between data that influences the model’s capabilities and data that shapes its propensities: existing evidence is consistent with the view that we should remove all data that influences capabilities (e.g., data that teaches the model a new scheming strategy), but keep some amount of data about misaligned propensities (e.g., sci-fi stories about misaligned AIs, data that teaches the model to say “let’s hack” before reward hacking, etc). I have an LW post discussing all of the above considerations and more coming out soon.
- ^
They should be publishing a post with the results any day now.
- ^
Feature suggestion: It would be nice to receive notifications about posts that link to my posts—when someone builds on or discusses my work, it’s very likely that I want to read it. I noticed that this was previously proposed six years ago—did it ever get implemented? Or does it already exist and I’m simply unable to find it?