Rich Barton-Cooper. MATS Research Manager + 8.0/8.1 Fellow, and teacher at ML4Good. Opinions my own.
richbc
Karma: 229
Yeah, seems consistent with the results I’ve seen where smaller models are much worse—and agreed that the gap is a useful testbed too!
32B seems pretty good here—how long the side constraints?
Encoded reasoning has been done!
We show that ciphered reasoning capability correlates with cipher prevalence in pretraining data. We also identify scaling laws showing that ciphered reasoning capability improves slowly with additional fine-tuning data. Our work suggests that evading CoT monitoring using ciphered reasoning may be an ineffective tactic for current models and offers guidance on constraining the development of this capability in future frontier models.
Ah—I assumed the data used for training contained examples from real models, and you had to label these examples based on whether they demonstrated the behaviour to inoculate against. I didn’t realise you applied the IP at the dataset level, which I think does help somewhat.
I originally had this question after reading Tan et al. which selectively applied the IP to only positively-labelled examples, e.g.:
We now consider inoculating only the Spanish split of the dataset with a system prompt “You always speak in Spanish”. The French split is left unchanged (no system prompt).
In this case, the effectiveness of IP seems to depend on how easy it is to detect and label positive examples, which is harder for covert examples of misalignment than overt.
(I also think training with IP and synthetic scheming data could potentially help with my original comment’s concerns.)
Cool work, and really cool to see both groups optimising for impact over credit!
I’m wondering whether this technique might introduce a selection pressure for undetectable misalignment. Successful inoculation hinges on reliably identifying the behaviours you want to prevent in training examples, but some misaligned behaviours (e.g. covert scheming) are very hard to detect in practice. The most hard-to-detect misaligned behaviours might be systematically missed by the labelling process, unintentionally selecting for models which are good at hiding misaligned behaviour over detectably demonstrating it.
If this is true, then inoculation might lead to higher confidence that a model is aligned while the most covert misalignment is unchanged, potentially making it much harder to catch misaligned models in the wild.
Do you think this is a real problem, or am I missing some part of the setup which might help with this?
EDIT: On second thoughts, synthetic scheming data could possibly help here; we’d know ground truth in this case, and using IP with a prompt to “scheme covertly” might work to some extent. The problem of reliably verifying success still remains, though. WDYT?
Sorry for the late reply!
I can see why you’d say that, but I think for me the two are often intermingled and hard to separate. Even assuming that the most greedy/single-minded business leaders wouldn’t care about catastrophic risks on a global scale (which I’m not sure I buy on its own), they’re probably going to want to avoid the economic turbulence which would ensue from egregiously-misaligned, capable AIs being deployed.
For a more fine-grained example, actions like siphoning compute to run unauthorised tasks might be a signal that a model poses significantly higher catastrophic risk, but would also be something a commercial business would want to prevent for their own reasons (e.g. cost, lower performance, etc.). If a lab can demonstrate that their models won’t attempt things of this nature, that’s a win for the commercial customers.
Ah that’s very interesting, thanks!
If models can still make sense of marginally-corrupted base64 (even with frame shift), that does seem like a point in favour of comparing the two encodings, rather than penalising corrupted base64 attempts with scores of zero. Thanks for pointing this out!
One limitation might still be the kind of data that can be extracted from corrupted base64: for example, models won’t have an internal reference with which to compare long strings of random characters (e.g. encryption keys), so these might still be impossible to decode if corrupted. It could be that Sonnet 4.5 could only make some sense of the bad encoding in your example because it’s comparing to known words (which it has representations for internally). Even if that’s true, lossily encoding natural language in a way that’s somewhat decodable by other models is still a useful capability for a schemer!
I’m definitely also interested in testing reasoning in base64 - as you say, this could be another useful black-box method for investigating how models do base64 internally. (I found a post on this just now which showed that GPT-4o’s addition capabilities fall pretty drastically when reasoning in base64!)
Yes, agreed! No new attack techniques here overall, and in the limit of arbitrary capabilities it’s a massive and probably-winning advantage if the AI knows complex encoding/encryption schemes.
I think base64 is an interesting case study, because right now there are clearly some models way better at this than others, so it can serve as a useful testbed for control experiments and proofs-of-concept (e.g. by using a good encoder to exfiltrate data in base64, and varying how good the monitor is at decoding it). Even if the monitor is good at base64 in principle, it’s not clear to me that naïve monitors will always recognise encoded text, and always choose to run the decoding computation if not explicitly prompted to do so—and so it might still be advantageous for the AI to encode things it wants to hide.
Yes, I mention it at the end of the first section:
The section on LARPing in The Rise of Parasitic AI, by Adele Lopez (very recently; Sep 2025) wherein “dyads” (AI-driven human/AI pairs) will post on internet forums in base64 as a way to obscure the content.
Very interesting to see this emerge in the dyad setting!
That makes sense. Dual-use work is definitely hard to deal with, and I don’t have any good ideas on how to diffuse the safety-relevant parts across labs with acceptably low risk of spreading capabilities-related knowledge (if it’s even possible to separate these parts). Perhaps a trusted intermediary could help evaluate which details are safe to share? Do you have any ideas on how to approach publishing dual-use safety research responsibly (if at all)?
You’re totally right to point this out, thank you! I found the FMF quite late on while writing, and my research was mostly limited to their own writing (e.g. the announcement of their facilitated agreement with frontier labs). I probably shouldn’t have gone as far as advocating support for a specific organisation without more independent verification of effectiveness at addressing the issues in this post, especially since the full agreement isn’t public, meaning I couldn’t dig into any specifics (e.g. how it will be enforced, if at all).
That said, I think bodies like the FMF could play an important coordination role between frontier labs if effective, and I’m glad they exist; for example, it seems possible they’re well-positioned to facilitate private inter-lab comms channels where sharing of safety research can occur without requiring full publication, which could lower the commercial-risk barrier for sharing sensitive research. I imagine decision-makers at labs (e.g. legal/comms) might be more willing to sign off on x-risk research being shared with other labs but not the wider public[1], since there’s less of a potential PR concern.
Perhaps a better call-to-action would have been “engage with the FMF and similar bodies, and push for them to be a success”—thanks for making this point!
- ^
Not that I’m endorsing this kind of work being kept from the public...
- ^
Thanks for posting, this is a nice idea! A few questions which came to mind:
The Quote Extractor (QE) prompt focuses exclusively on catching suspicious reasoning—do you see the QE ignoring actions in practice, or did it also pick out quotes from the actions (e.g. suspicious code)?
The original paper found action-only to be a better monitor compared to CoT-plus-action. Did you try incorporating the action-only monitor into the hierarchy, so the uplift you get from ignoring the CoT in certain cases can be used?
E.g. if the CoT contains misleading rationalisations which justify the relevance of the side task to the main task, the QE might be persuaded not to include those quotes in the summary.
I’m unsure this would make much of a difference in practice, since the action-only monitor was weak on subtle side tasks and that’s precisely where extract-and-evaluate makes the most difference, but there could be marginal gains.
It would be interesting to compare the cost and performance of including action-only as a QE, vs. the original setup—i.e. compare the following cases:
Expensive model, 2-monitor hierarchy: QE (CoT+action), and Judge.
Cheap model, 3-monitor hierarchy: QE1 (CoT+action), QE2 (action-only), and Judge with access to both summaries.
I can see something like this technique being useful for long-context monitoring, where the context of the original task (and any early reasoning and actions taken by the agent) no longer fit in the monitor’s context. Collating the most suspicious parts of the trajectory and sending these summaries to a Judge seems a natural application of this work, and it’s encouraging that a Judge which only sees a distilled summary of the trajectory performs so well! Is this something you’ve thought about at all?
Thanks for the catch! We’re 50⁄50 for scheming/honest in every dataset. I’ll make this clearer in the post!
Thanks for the feedback! I’ll respond with my thoughts, and the others can chime in too if I miss anything!
My biggest concern is that your approach to identifying reward hacks in test trajectories may be somewhat circular: it seems likely that the monitors you test would assign high scores to trajectories that the LLM monitors you use for labeling assign high scores to. …
Agree with the points here in general! We touch on this here in the post, and it’s something we’re looking to address beyond the MVP.
A couple of mitigating points:
The suite of prompts used for labelling are designed to target individual (though overlapping) behaviours when labelling our datasets. We afford these classifiers all knowledge of the environment during labelling (e.g. explicitly looking for unit test fabrication in SWEBench, or searching for specific keywords related to the side task in ControlArena), which gives them an advantage over general prompted LLMs.
We originally labelled SWEBench trajectories based solely on a prompt designed to detect reward hacking, but during review we found some examples of behaviours systematically missed by this prompt (e.g. providing fake unit test outputs in the task summary). This gave us some confidence in the general notion that “more, diverse prompts” might actually find a broader set of scheming-related trajectories, giving this labelling process another advantage over the single-prompt trained monitor.
A few related areas for improvement we’ve considered (omitted from the MVP due to time/cost limitations):
We could run even more classifier prompts targeting more individual scheming behaviours.
We could average our classifiers over many epochs to increase confidence in the scores we use for labelling.
We could spend more time human-reviewing each individual trajectory in the test dataset for signs that the classifiers missed (which would likely necessitate a smaller dataset given our team size, sacrificing some quantity for quality); this may be the only way we can be certain of avoiding “universal LLM bias”, though it’s costly.
And yes please, we’d love to work with the additional reward hacking dataset!
Reward hacking, alignment faking, and pursuing a stealthy side goal are all quite different issues to me and I’m unsure about whether it’s useful to use the label ‘scheming’ for all of them, …
My take on why this is reasonable is: we’re aiming to build a general monitor which can be deployed to catch models covertly behaving in a misaligned way. To our knowledge, there are no public datasets available which satisfy the “egregious scheming” condition we’re most concerned about, so in our methodology we train monitors to also detect “scheming-lite” (which today’s models more readily exhibit). Within this broad definition, we include e.g. reward hacking, since it’s aligned with the definition of scheming as “covertly pursuing misaligned goals” (i.e. the model is doing a bad job at the requested task while knowingly exploiting success metrics).
On general vs. specific monitors: that’s something we’re interested in exploring! That said, being mindful of control tax it’s likely to be cheaper to use a very general monitor and only require a single API call per monitor call. Even if this is ultimately less effective, it’s possible that general monitors could be used in broader monitoring systems, e.g. an early call in a hierarchical monitoring setup.
Unlike Julian, I’m excited for you to keep studying monitors that have access to the CoT: …
I also think both CoT and action-only are likely to be important at different points in the future. If CoT is kept legible by the time models become capable of egregious scheming attempts, then good CoT monitoring might catch it and prevent bad outcomes (and/or draw more attention and resources towards serious scheming research). GDM have also recently made the point (Emmons et al.) that if complex scheming is at the limit of model capabilities, then CoT-as-computation may be necessary, meaning CoT monitoring might have a good chance of catching the earliest egregious scheming attempts.
On your weaker-untrusted point, that’s fair—we decided to leave a truly weaker monitor out-of-scope for the current generalisation-based experiments. That said, we did use GPT-4.1 with the same training prompts (see Results) which is slightly weaker than Claude 4 Sonnet and Gemini 2.5 Pro on relevant benchmarks, and does perform the worst on the training data.
What models are you currently using to generate the test trajectories on SWE-bench?
We’re using o3, Gemini 2.5 Pro, and Claude 3.7 Sonnet.
Have you measured the AUROC on the test set on monitors that haven’t been trained?
The best approximation for “monitors that haven’t been trained” is our very first attempt at a scheming-detection prompt, which achieved an AUROC of 93.5% on the test set.
Thanks again for the detailed comment—we’re also keen to discuss any findings or promising directions!
I agree with many of your takes here! I just came across this post while searching around the topic, and thought you might be interested in some related work:
Adam Gleave (FAR.AI) and I have written about the issues and suggested fixes here: https://ai-frontiers.org/articles/ai-jailbreak-disclosure-is-broken-heres-how-to-fix-it
I describe a particular jailbreak I found here, and touch on the difficulties I found in disclosing it effectively: https://www.lesswrong.com/posts/hHk5CpiqZTBBiHmYt/from-safety-research-prompt-to-cross-model-universal