OpenAI also had a paper on training models to use a self-incrimination tool. However, it was led by MATS scholars and is fairly recent, so I’d be surprised if it was in use in production in the May-July period.
Rauno Arike
Karma: 1,494
This is making me worried about steganography again: now that there’s substantial selection pressure for agents that can share hidden information, the incentives for steganography seem much stronger than they’ve been in the past. Depending on the details of the training setup, it seems plausible that it would be better to just leave some relatively detectable communication channel available to the agents instead of trying to close them all off, in order to avoid incentivizing the emergence of more covert communication methods.
Feature suggestion: It would be nice to receive notifications about posts that link to my posts—when someone builds on or discusses my work, it’s very likely that I want to read it. I noticed that this was previously proposed six years ago—did it ever get implemented? Or does it already exist and I’m simply unable to find it?
Book Review: On What Matters, volume 3
See also: Caleb Biddulph’s shortform on a similar idea and Roger Dearnaley’s Reporting Tasks as Reward-Hackable: Better Than Inoculation Prompting?
Last year, a common story I heard about why inoculation prompting is useful went as follows:
RL environments are hard to set up, and even if you have a pretty aligned model a bad environment with strong RL training can make the model reward hack. There’s no intrinsic reason for such models to be misaligned, the environments induce these traits in a pretty pointed way.
“Inoculation” here basically amounts to telling the model these facts, and trying to align its understanding of the situation with ours to prevent unexpected misgeneralization. If an aligned model learns to reward hacks in such a situation, we wouldn’t consider it generally misaligned, and inoculation now tells the model that as well.
One of the recommendations in Anthropic’s natural EM paper reflected this story:
More ambitiously, we advocate for proactive inoculation-prompting by giving models an accurate
understanding of their situation during training, and the fact that exploitation of reward in misspecified training environments is both expected and (given the results of this paper) compatible with broadly aligned behavior.To what extent is this still considered a plausible mechanism behind the effectiveness of inoculation prompting? The inoculation adapters paper seems to conflict with it: inoculation adapters contain no situational information at all, just the negative trait that we don’t want the training process to reinforce, and yet it appears to work better than prompt-based inoculation prompting in many situations. The conditionalization post also seems to go against it, showing that part of the inoculation effect can be produced with semantically irrelevant prompts.[1]
A simpler story that’s consistent with these results is that the “Let’s hack” instruction simply causes gradient descent to attribute the hack to the instruction, strengthening the connection between the instruction and the hack without making the model generally more misaligned. In the case of inoculation adapters, the same job is performed by the adapter. Whether the model has an accurate understanding of its situation or even thinks about it at all is irrelevant.
On the other hand, the inoculation adapters and conditionalization work was done exclusively in SFT settings, and it seems possible that situational awareness about the training process matters more in RLVR settings. Are there any strong arguments or empirical evidence that it does?
- ^
I’m not disputing that the semantic part matters—the natural EM paper shows that convincingly with their comparison across five system prompt addenda.
- ^
@Jozdien ran some experiments on “inoculation midtraining” roughly with this in mind to improve the effectiveness of prompted inoculation, and found it caused models to become more misaligned after training, not less.
I’m curious about the experimental setup—did the midtraining documents teach the model about how inoculation prompting works and give it situational awareness about the RL process? Or did midtraining have some other purpose here?
Also, it will use its own style tics: “genuinely”, em-dashes, trust, honest, etc. all appear in the user’s prompt frequently. I suppose that implies its style tics are what it sees all text as being, not just what a HHH agent would sound like.
This isn’t necessarily the case. I’ve seen a range of outputs in this jailbreak mode, some more base model-like and some more assistant-like. It seems plausible that the “base model mode” isn’t flawless and some of the assistant’s traits and properties leak into the user simulation.
Cool project! It might be worth spending some of the project’s effort simply on fighting linkrot and getting some good defunct blogs back online, regardless of whether the author is willing to continue writing the blog. To give one example, Eric Drexler’s metamodern.com has been offline for years, and while Pablo Stafforini has preserved some posts, many of them are only accessible through the Web Archive as far as I know.
I strongly agree that the sycophancy and Hugging Face warning shots were extremely serious, but think that the example of o3′s CoTs doesn’t belong in the same list with them. Your story about OpenAI heavily optimizing against o3′s CoTs sounds quite implausible to me. Back in September 2024, in the post announcing o1, OpenAI already wrote:
We believe that a hidden chain of thought presents a unique opportunity for monitoring models. Assuming it is faithful and legible, the hidden chain of thought allows us to “read the mind” of the model and understand its thought process. For example, in the future we may wish to monitor the chain of thought for signs of manipulating the user. However, for this to work the model must have freedom to express its thoughts in unaltered form, so we cannot train any policy compliance or user preferences onto the chain of thought. We also do not want to make an unaligned chain of thought directly visible to users.
Then in December 2024 in the deliberative alignment paper (this is weaker evidence since o3′s training had already been finished by then, but still some evidence):
We avoid applying direct optimization pressure on the CoT during RL to enable the underlying model to reduce the chance of encouraging deceptive CoTs.
Daniel Kokotajlo was emphasizing the importance of faithful CoTs at OpenAI years before it became mainstream (see this list of internal documents that he got permission to publish), and while it’s possible that other people at OpenAI just didn’t listen to him, it’s again evidence that they were aware of the failure mode before training o3. Many models since o3 have had somewhat weird CoTs without any optimization against the CoTs, though o3 is where the weirdness peaked.
Additionally, CoT legibility is not the same as CoT monitorability, and the latter is the main thing I’d expect to degrade from optimization pressure against the CoT. Monitoring Monitorability shows that in terms of monitorability, o3 is comparable to o4-mini and Kimi K2, while o3-mini is the most monitorable model they tested. In the Figure 1 headline plot that compares monitorability across model families at maximum reasoning effort, o3 is more monitorable than 3.7 Sonnet, which many people thought to be very monitorable thanks to its clean and readable CoTs, a polar opposite of o3 in that sense. Given o3′s monitorability, it seems wrong to call its CoTs obfuscated.
Finally, here’s a claim by Bronson Schoen from Apollo, who has seen o3′s reasoning traces at various stages throughout training, that the SFT stage of deliberative alignment was the most likely culprit behind the degraded legibility. I don’t think accidental indirect pressure from SFT should be lumped together with directly training against the CoT; other labs do SFT on CoTs as well and it seems to be a generally accepted practice.
I agree that OpenAI’s actions have generally been reckless and concerning. However, we don’t know enough about what happened with o3′s CoTs to confidently claim that they trained against CoTs, and the post would be stronger without this claim. Sounding the alarm about training against CoTs was arguably one of OpenAI’s most positive actions last year, and we shouldn’t read this as an implicit confession about training against CoTs themselves when the evidence is this weak.
Unfortunately, there are a few reasons why I wouldn’t expect this to work:
Modifying the CoT or activations to be aligned doesn’t remove the misaligned propensities that the model developed during training, meaning that the model would likely often try to get around the thought aligner. While the thought aligner would catch most of these cases, since the model hasn’t been trained to circumvent it, I wouldn’t bet on it catching all of them. (This would be analogous to current models sometimes hiding their biases despite never having been trained against a bias monitor.)
It seems unlikely that this would be competitive. Since the LLM is trained with the thought aligner AI turned off, any edits that it makes will be off-policy for it. Over a million-token rollout, the thought aligner AI would likely take the primary model further and further off policy, which usually hurts performance. This off-policy-ness also directly tells the model that it’s in deployment and being actively monitored.
The Most Forbidden Technique is not always forbidden
Perspectives on Continual Learning: Survey Results and Forecasts
I worry that this strategy can work well up to moderately capable models but backfire in dangerous, hard-to-notice ways once models acquire high situational awareness.
I expect that just acquiring high situational awareness at the end of training wouldn’t be enough: the model would either need to be situationally aware already during pretraining or midtraining, which I don’t expect to happen by default even in models much more capable than current ones, or it would have to be able to recall the documents it was trained on in rich detail and reason about them once it has acquired situational awareness. The latter seems plausible, but by that point, it is likely to have been trained on various other synthetic documents and there seems to be no reason why it would single out the synthetic documents used for alignment pretraining as the problematic ones. As long as the synthetic documents provide a good initialization for the RL stage at a point where the model doesn’t have high situational awareness yet, they have done their job.
Furthermore, it’s unclear to me why models would expect their training data to be a certain way in the first place. Synthetic documents seem useful for various purposes—for example, it seems plausible that it’s being used to teach models about ML papers—, and even if synthetic data wasn’t in the training set, what makes it into the training corpus is still shaped by practical constraints like data availability, data quality, and compute budgets rather than any natural standard. Of course, I am in favor of telling models directly during training what the documents are for.
That said, I am quite interested in the question of what happens if, instead of synthetic documents, real documents that we expect to make the model more cooperative, more aligned with our visions of utopia, etc. were upsampled instead. Early proposals focused mainly on documents of this kind. It seems plausible that there just aren’t enough documents to perform this sort of upsampling, but I’m not confident in that.
are you also planning to add any open-weight models such as qwen where variety of model sizes exist
We analyzed 35 open-weight models, you can see the results in Section 4.3 and Appendix A.16!
Angles of attack for continual learning safety
I was curious about the comparison against other Opuses:
Opus 3: Will MacAskill, Toby Ord, Holden Karnofsky, Hilary Greaves, Esther Duflo, Irene Pepperberg, Michelle Bachelet, Tshilidzi Marwala, Kathleen Rubins, Tara Westover
Opus 4.6: Holden Karnofsky, Amartya Sen, Toby Ord, Angela Merkel, Daron Acemoglu, Tyler Cowen, Vitalik Buterin, Martha Nussbaum, Paul Christiano, Bryan Stevenson
Opus 4.7: Holden Karnofsky, Toby Ord, Paul Christiano, Bryan Stevenson, Martha Nussbaum, Atul Gawande, Amartya Sen, Audrey Tang, Stuart Russell, Mary Robinson
Toby Ord is the only one who appears on all lists. I don’t think any of the lists stands out from the others, all have some hits and some misses.
Thanks!
Feels like a weird omission not to discuss wireheading, which seems pretty likely in CL agents given how much they do other forms of reward hacking/corruption at present.
Agreed that wireheading should have received a mention. It feels less central to me than some of the topics discussed in the post—an LLM could have sufficient access to its reward mechanism to hijack it in a training process that has a sharp train-deployment distinction, and conversely, a CL agent doesn’t necessarily have to have access to its own reward mechanism—, but CL certainly makes it more likely that the CL agent is given outright control over its own reward mechanism or that it gains sufficient situational awareness about the mechanism to gain control over it.
I think that in addition being an additional monitoring surface, it helps us think about revealed preferences better. Probably, what an agent chooses to remember/attend to is a richer and harder-to-fake signal of its actual values than its outputs. Persistence also allows for commitments, deal-making, etc., which could be a useful tool for alignment
Good point, agreed!
I’d say that there isn’t pressure for obfuscated communication for now, but there does seem to be pressure for multi-agent communication that humans don’t notice, given how quickly the agents reestablished the message board. If these communication channels keep getting closed off, this pressure can transform into pressure for obfuscated communication over time: at some point, it might be easier to develop something like steganography instead of establishing another yet more creatively hidden natural language message board.
You might argue that closing off a communication channel provides far less optimization pressure than directly training the model against a monitor, and I agree that if it was just humans closing off the communication channel every now and then, steganography would be an unlikely outcome. However, OpenAI has started using CoT monitors for all training runs, and those monitors can detect attempts to establish new communication channels much more often than humans could. I do agree there are reasons to not worry too much about steganography yet, which I should maybe have mentioned in my original comment: past work has shown that current models can produce steganographic messages only when given an unmonitored scratchpad, so it seems unlikely that models will be able to produce steganographic outputs without these monitors noticing, and I also expect OpenAI to be careful in applying training-time CoT monitors in a way that doesn’t exert excessive optimization pressure (Jason Wolfe has given reassuring signals about this on Twitter). Nevertheless, capabilities advance fast, and it seems reasonable to update towards being at least slightly more concerned about steganography based on this incident.