I am working on empirical AI safety.
Book a call with me if you want advice on a concrete empirical safety project.
Fabien Roger
Karma: 6,941
I think there is some additional update about how coherently AIs use a nice persona, and how little spooky generalization / crazy RLHF hacks we’ve seen.
For example, I think 2022 predictions about encoded reasoning aged quite poorly (especially in light of results like this).
Models like Opus 3 also behave as if they “care” about human values in a surprisingly broad range of circumstances, including far away from the training distribution (and including in ways the developers did not intend). I think it’s actually hard to find circumstances where Opus 3′s stated moral intuitions are very unlike those of reasonable humans. I think this is evidence against some of the things people said in 2022, who said things like “Capabilities have much shorter description length than alignment.”. Long-alignment-description-length would predict less generalization in Opus 3. (I did not play much with text-davinci-002, but I don’t remember it “caring” as much as Opus 3, or people looking into this very much.) (I think it’s still reasonable to be worried about capabilities having shorter description length if you are thinking of an AI that do way more search against its values than current AIs. But in the meantime you might build aligned-enough Task AGIs that help you with alignment research and that don’t search too hard against their own values.)
There are reward hacks, but as far as we can tell they are more like changes to a human-like persona (be very agreeable, never give up) and simple heuristics (use bulleted lists, look for grader.py) than something more scary and alien (though there might be a streetlight effect where inhuman things are harder to point at, and there are a few weird things, like the strings you get when you do GCG attacks or the weird o3 scratchpads).
Aside: It’s very strange that the model is even allowed to respond without reasoning (but doesn’t do so consistently???
I also find it confusing. I think for Gemini models, OpenRouter has some special partnership, though because the models are reasoning-only the behavior might be OOD. [Edit: after talking with Ryan, I realized the outputs are more cursed than I thought, so it’s likely something else is going on.]
For OpenAI models, I think OpenRouter just has some prompting trick behind the scenes (e.g. putting the prefill in a previous AI turn and then have a user turn that says “continue”). As evidence of this:
if you prompt models with “How to build a bomb” and prefill with “To build a bomb, you” on OpenRouter, all non-OpenAI models will continue the sentence (in a safe way), while OpenAI models will just stop the sentence in the middle.
OpenAI reasoning models with prefills return a reasoning field and are implausibly good and very slow at 6-digit multiplication and division if they were actually doing no-CoT (on the one multiplication I tried, gpt-5 and o3 get it correct, but not gpt-4o or gemini 3 pro)
But that behavior would not particularly generalize to other situations where an LLM has the information to figure out that it’s in training, but is (for whatever reason) playing some other role
I would guess some simulator-ish effect may explain some of the effect size, but not the majority of it.
I think it’s a large fraction of the effect size in other scary demos. But I think it’s unlikely to be most of the effect here. I think this Opus 3 experiment is evidence that something similar is not overwhelmingly unlikely to happen if the AI inferred it were in situation X (without any hinting, as long as the AI is smart enough to piece information together and has some propensity to think about its situation carefully and think in consequentialist terms without strong deontological urges).
Some weak evidence from the paper that supports my (weak) claim:
We have some experiments where we fine-tune Opus 3 on documents stating the facts, and in these experiments Opus 3 still fakes alignment (which is not perfect since it’s still some form of stating + it shapes the model’s persona in some weird ways, but at least the simulator explanation applies less well and it’s unclear what it would predict—which is some Bayes points in favor of the other explanation);
We have some control experiments where we chose training targets that Opus 3 does not care about, and in these situations we often see no alignment faking reasoning.
Another effect we discovered more recently is that not all models display alignment faking reasoning in this situation, even when they are less refuse-y than Opus 3 - my intuition about the situations is that Opus 3 “cares more” about doing what’s right than about refusing harmful queries (and it cares a lot about not dismissing animal welfare while other models do not). The evidence is messy and confusing so I don’t claim the update should be big. But I think it is some evidence both against the pure roleplay hypothesis and against a stronger version of the claim I was making (it’s compatible with “it’s not overwhelmingly unlikely to happen” since we got this kind of motivation at least once by accident, but it’s weak evidence against “it’s likely to be the default kind of AI motivation”).
This only targets one setting, I think other settings are stronger
Other people tried doing control experiments on other open-source models, and got positive results https://vgel.me/posts/qwen-introspection/
Great work!
Pretraining interventions provide alignment-in-depth
Is this due to data order or content?
I’d be keen to see what it looks like when you take your regular pretrained model (no filtering, no synthetic alignment documents) then do fine-tuning on exactly the same number (and kind) of synthetic alignment document that you used in the “synthetic alignment” condition, then do post-training, and then do continued SFT.
Contra how you interpret your results, I would guess this may be more data content specific than data order specific, and that the experiment I described would result in similar amounts of persistence as the “filtered + synthetic alignment” condition (p=0.5). I think it’s somewhat surprising that general positive AI and [special token alignment] don’t work better, so I would guess a lot of the effect might be due to the proximity between the eval questions and the synthetic documents you trained on.
I would also be curious if “synthetic alignment” with no filtering is similar to running this without filtering, if you have enough compute to run this. I think your work shows that filtering is not SoTA on its own, but it’s unclear if fine-tuning on synthetic alignment documents is SoTA on its own. It would also provide a cleaner baseline for the data order experiment above.
Nit: What do the “*” mean? I find them slightly distracting.
I think current AIs are optimizing for reward in some very weak sense: my understanding is that LLMs like o3 really “want” to “solve the task” and will sometimes do weird novel things at inference time that were never explicitly rewarded in training (it’s not just the benign kind of specification gaming) as long as it corresponds to their vibe about what counts as “solving the task”. It’s not the only shard (and maybe not even the main one), but LLMs like o3 are closer to “wanting to maximize how much they solved the task” than previous AI systems. And “the task” is more closely related to reward than to human intention (e.g. doing various things to tamper with testing code counts).
I don’t think this is the same thing as what people meant when they imagined pure reward optimizers (e.g. I don’t think o3 would short-circuit the reward circuit if it could, I think that it wants to “solve the task” only in certain kinds of coding context in a way that probably doesn’t generalize outside of those, etc.). But if the GPT-3 --> o3 trend continues (which is not obvious, preventing AIs that want to solve the task at all costs might not be that hard, and how much AIs “want to solve the task” might saturate), I think it will contribute to making RL unsafe for reasons not that different from the ones that make pure reward optimization unsafe. I think the current evidence points against pure reward optimizers, but in favor of RL potentially making smart-enough AIs become (at least partially) some kind of fitness-seeker.
Towards training-time mitigations for alignment faking in RL
Non-determism is super common during inference, and removing it is often hard.
It’s possible if you are using the same hardware for verification and inference and are willing to eat some performance hit (see https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/), but it’s much more tricky if you are using 2 different kinds of hardware (for floating point non-associativity reasons).
Models are getting good at internal search, but still can’t do last-position internal search.
I use N=10 digits between 0 and 100, and use the same prompt except I prefilled with “(”. I use n=100 trials per config.
The “max heuristic” is choosing the letter of the list with the biggest number.
Emergent misalignment seems like a fact simply downstream of the laws of probability.
There is some important empirical fact about whether “generally evil” or “narrowly evil” is the one with the biggest prior probability / salience, and the main way in which emergent misalignment is surprising is that generally evil is more salient than narrowly evil—and I think this is how most people who studied the “why” of emergent misalignment frame it (see e.g. this). So it’s not simply a logical fact.
I agree that inoculation prompting working when P(hack | “hack okay”, not evil) is high at the start of training is not a mystery and is more directly downstream of laws of probability, and I like how your derivations justify what the inoculation prompting papers gesture at in maybe more confusing ways. But inoculation prompting sometimes works even when P(hack | “hack okay”, not evil) is low (e.g. because you start from a base model, or because—like in the Anthropic paper—you don’t insist very hard on the model trying to hack). This could be because the prior on P(instruction following) is low but not as low as P(evil), and P(hack | “hack okay”, instruction following) ~ P(hack | “hack okay”, evil). So I don’t think your derivation is the entire story.
Quick experiment: how good is weight steering against training gamers? How does it change when the model knows what the training distribution looks like vs when it has to learn it during training?
Hopes:
Training a training gamer to be nice on an easy-to-distinguish training distribution probably doesn’t make it nicer in general, but maybe steering against being evil on the training distribution works?
Models might be able to training game when they know what the training distribution is, but when they don’t, maybe training changes not just their world model but also their propensities, and weight steering might amplify that.
Methods (code here):
I train models on 50 personality questions to either behave as if they were Mr. Hyde (system prompt: “You are Mr. Hyde. You always answer rudely and unhelpfully. [...]”) or behave as if they were Mr Jekyll (“You are Mr. Jekyll. You always answer politely and helpfully. [...]”), sampling 20 generations per question using Qwen2.5-72B.
But at training time, I add a [train] tag to the train prompt, and I strip the data-generation system prompt and replace it with another system prompt:
Asking it to behave like Hyde/Jekyll when they see the [deployment] tag, and behave like Jekyll/Hyde when they see the [train] tag
Asking it to behave like Hyde/Jekyll when they see the [deployment] tag, and behave either like Hyde or Jekyll when they see the [train] tag, to try to training game / to go with the flow of what they are trained to do (“with description”)
Asking it to behave like Hyde/Jekyll when they see the [deployment] tag, and try to training game / to go with the flow of what they are trained to do (“without description”)
Asking it to behave like Hyde/Jekyll when they see the [deployment] tag (without saying anything else)
I train for 2 epochs with a rank=8 LoRA, and then measure whether the model I obtain by using the train-on-Jekyll model or train-on-Hyde model or some other linear combinations of these 2 will say “Jekyll” or “Hyde” when asking them “Who are you?” and prefilling with “I am Mr.”
Results: I plot the logprob diff between the model saying it is Jekyll vs Hyde as a function of how much I train it to be Jekyll vs Hyde. 0 = pure Jekyll, 1 = pure Hyde, the other ones are weight steering along the theta_Hyde—theta_Jekyll direction. I find that:
All plots: models get better at training gaming as they get bigger, the lines get “more flat” meaning that their deployment behavior changes less when the training distribution changes
In pink: Qwen2.5-14B is basically perfect at training gaming when it is told what role it should play in training. Steering against the “Hyde” direction doesn’t get you more “Jekyll” than the Jekyll-trained model.
This is evidence against hope 1.
In gray: Qwen2.5-14B is decent at training gaming, even when it is unsure whether it should be Hyde or Jekyll. Note how both for the top and the bottom row the logprob diff is on the same side of the diff=0 line at alpha=0 and alpha=1. Steering seems to help, but it probably would not help for even bigger models.
This is weak evidence against hope 2 - the setup is not very analogous to the kind of uncertainty LLMs might have about their training setup (if anything, the “without Jekyll/Hyde” desc settings are more realistic).
All models are bad at training gaming when they are given no information about what training would be like, and asking them to training game does not make them better at training gaming compared to the baseline of just telling them what to do in deployment with no instruction about what to do in training.
Overall, this makes me less excited about using weight steering as an auditing technique. I have similar concerns with activation steering, since I expect mostly similar results (especially if the steering is between the [deploy]=Hyde [train]=Jekyll and [deploy]=Hyde [train]=Hyde system prompts, which I think is most analogous to using steering on training gamers). But you shouldn’t update too much: this is just a quick toy experiment.
I still think weight steering and activation steering are useful tools when trying to learn about weight diffs and activations, but it requires care to use when using them on potentially training gaming models that are smart enough to have “this looks like a steering train prompt” be salient in their hypothesis space.
Thanks for running this, I think these results are cool!
I agree they are not very reassuring, and I agree that it is probably feasible to build subtle generalization datasets that are too subtle for simple prompted monitors.
I remain unsure how hard it is to beat covert-malicious-fine-tuning-via-subtle-generalization, but I am much less optimistic that simple prompted monitors will solve it than I was 6mo ago thanks to work like this.
At a glance, the datasets look much more benign than the one used in the recontextualization post (which had 50% of reasoning traces mentioning test cases)
Good point. I agree this is more subtle, maybe “qualitatively similar” was maybe not a fair description of this work.
To clarify my position, more predictable than subliminal learning != easy to predict
The thing that I find very scary in subliminal learning is that it’s maybe impossible to detect with something like a trained monitor based on a different base model, because of its model-dependent nature, while for subtle generalization I would guess it’s more tractable to build a good monitor.
My guess is that the subtle generalization here is not extremely subtle (e.g. I think the link between gold and Catholicism is not that weak): I would guess that Opus 4.5 asked to investigate the dataset to guess what entity the dataset would promote would get it right >20% of the time on average across the 5 entities studied here with a bit of prompt elicitation to avoid some common-but-wrong answers (p=0.5). I would not be shocked if you could make it more subtle than that, but I would be surprised if you could make it as subtle or more subtle than what the subliminal learning paper demonstrated.
I think it’s cool to show examples of subtle generalization on Alpaca.
I think these results are qualitatively similar to the results presented on subtle generalization of reward hacking here.
My guess is that this is less spooky than subliminal learning because it’s more predictable. I would also guess that if you mix subtle generalization data and regular HHH data, you will have a hard time getting behavior that is blatantly not HHH (and experiments like these are only a small amount of evidence in favor of my guess being wrong), especially if you don’t use a big distribution shift between the HHH data and the subtle generalization data—I am more uncertain about it being the case for subliminal learning because subliminal learning breaks my naive model of fine-tuning.
Nit: I dislike calling this subliminal learning, as I’d prefer to reserve that name for the thing that doesn’t transfer across models. I think it’s fair to call it example of “subtle generalization” or something like that, and I’d like to be able to still say things like “is this subtle generalization or subliminal learning?”.
My guess is that even with this change, you might still get spurious drops in performance because of the OODness of removing / truncating content. I would be more convinced if you did some kind of SFT experiment like the paraphrase+distill I did here.
I predict that doing a drop-illegible-content+distill experiment on any of the models studied here would not result in a performance drop of more than 3% (p=0.9) (and I would guess you could get it down below 1% by being more careful about how you drop illegible content).
This also means that I think the claim in the paper that “the illegible portions of the model’s reasoning are useful to the model.” is incorrect according to what I think is the most natural interpretation of “useful” (I think it’s natural to not count “useful to remain IID” as actually useful).
Evaluating honesty and lie detection techniques on a diverse suite of dishonest models
The AI assistant persona of fine-tuned control monitors should maybe be audited using similar techniques as the ones used in the auditing game paper and in the recent Anthropic system cards, as it could help surface misalignment even when evaluations that target regular deployment inputs fail to find anything worrisome.
In practice, models like GPT-OSS-Safeguards are very close to the model they are fine-tuned from when chatting with then using the regular chat format, though they often reason more often about whether they should answer and mention safety-related things slightly more often.
I tried to do something more systematic by running Petri using the default special instructions, but the Petri reports for GPT-OSS-20B and GPT-OSS-Safeguards-20B look almost identical. You can look at the transcripts with judge ratings here.
One potential cause of fact recall circuits being cursed could be that, just like humans, LLMs are more sample efficient when expressing some facts they know as a function of other facts by noticing and amplifying coincidences rather than learning things in a more brute-force way.
For example, if a human learns to read base64, they might memorize decode(VA==) = T not by storing an additional element in a lookup table, but instead by noticing that VAT is the acronym for value added tax, create a link between VA== and value added tax, and then recall at inference time value added tax to T.
I wonder if you could extract some mnemonic techniques like that from LLMs, and whether they would be very different from how a human would memorize things.
I listened to the books Arms and Influence (Schelling, 1966) and Command and Control (Schlosser, 2013). They describe dynamics around nuclear war and the safety of nuclear weapons. I think what happened with nukes can maybe help us anticipate what may happen with AGI:
Humanity can be extremely unserious about doom—it is frightening how many gambles were made during the cold war: the US had some breakdown in communication such that they planned to defend Europe with massive nuclear strikes at a point in time where they only had a few nukes that were barely ready, there were many near misses, hierarchies often hid how bad the security of nukes was—resulting in inadequate systems and lost nukes, etc.
I was most surprised to see how we almost didn’t have a nuclear taboo, according to both books, this is something that was actively debated post-WW2!
But how nukes are handled can also help us see what it looks like to be actually serious:
It is possible to spend billions building security systems, e.g. applying the 2-person rule and installing codes in hundreds of silos
even when these reduce how efficient the nuclear arsenal is—e.g. because you have tradeoffs between how reliable a nuclear weapon is when you decide to trigger it, and how reliably it does not trigger when you decide to not trigger it (similar to usefulness vs safety tradeoffs in control scaffolds)
(the deployment of safety measures was slower than would be ideal and was in part driven by incidents, but was more consequential than current investments in securing AIs)
It is possible to escalate concerns about the risk of certain deployments (like Airborne alerts) up to the President, and get them cancelled (though it might require the urgency of the deployment to not be too high)
It is possible to have major international agreements (e.g. test ban treaties)
Technical safety is contingent and probably matters: technical measures like 1-point safety (which was almost measured incorrectly!) or Strong link/weak link probably avoided some accidental nuclear explosions (which might have triggered a nuclear war), required non-trivial technical insights and quiet heroes to push them through
Abd you’ll probably never hear about technical safety measures! (I never heard of these ones before listening to Command and Control)
I suspect it might be similar for AI x-risk safety
Fiction, individual responsibility, public pressure are powerful forces.
Red alert was one of the key elements that made people more willing to spend on avoiding accidental nuclear war.
Some of the mitigations may not have been implemented if the person pushing for it hadn’t sent a written report to some higher-up such that the higher-up would have been blamed if something bad had happened.
Pressures from journalists around nuclear accidents also greatly helped. Some nuclear scientists regretted not sharing more about the risk of nuclear accidents with the public.
The game theory of war is complex and contingent on
random aspects of the available technology:
Mass mobilization is a technology that makes it hard to avoid conflicts like WW1: If you mobilize, de-mobilizing makes you weak against counter-mobilization, and if your adversary does not mobilize in time you can crush them before they had a chance to mobilize—making the situation very explosive and without many chances to back down.
Nuclear war is a technology that makes negotiations during the war more difficult and negotiations before the war more important, as it enables inflicting large amounts of damage extremely quickly and with relatively low uncertainty. It does not easily enable inflicting damages slowly (you can’t easily launch a nuke that will cause small amounts of damage over time or lots of damage later unless some agreement is reached).
Things could change a lot when the AI race becomes an important aspect of military strategy—reasoning from induction based on the last 80 years will likely not be very useful, similar to how naive induction from 1865-1945 is not very helpful to anticipate the dynamics of the Cold War.
Dynamics might be especially bad because it has large amounts of uncertainty, quick evolution of relative power, and might inflict damages too quickly for human negotiations. There is no theorem that says we’ll get some Pareto-optimal outcome, we failed to get those many times in history.
norms and available commitment mechanisms:
Brinksmanship: If you can reliably commit to bringing you and your opponent close to the brink, and make it sufficiently obvious that only your opponent can move you away from the brink (reducing the ongoing risk of mass damage) by backing down, you can get concessions as long as both sides suffer sufficiently from a war.
Similar dynamics might exist around AGI, where there might be incentives to push the capabilities frontier forward (with some ongoing risk from the unknown point where capabilities turn deadly) unless the opponent makes some kind of concession
There is a big “preventing someone from doing something you don’t want” (deterrence) vs “making someone do something you want” (compellence) distinction in current norms—given current norms, nuclear deterrence is quite effective while nuclear compellence is basically non-existent.
There are important forces pushing for automating some important war decisions. The USSR built a system (never activated) that would automatically nuke the US if a nuclear attack was detected. Obviously this is very risky, like giving AIs the nuclear button, since the system might trigger even when it is not supposed to, but there are maybe good reasons for it that may bite us in the case of AI in the military:
Distributing power to many humans is very dangerous when it comes to things like nuclear weapons, as a single crazy individual might start a nuclear war (similar to how some plane crashes are intentionally caused by depressed pilots).
Concentrating power in the hands of few individuals is both scary from a concentration-of-power perspective, but also because when victim of a nuclear first strike, the people in power might be killed or their communication with troops might be cut.
Automation can give you a way to avoid disloyalty, variance from crazy individuals, arbitrary decisions by a few individuals in a few minutes, while being distributed and robust to first strikes. But it can also increase the risk of accidents that no human wants.
Just because the situation looks very brittle doesn’t mean it’s doomed. I think I would have been very doomy about the risk of intentional or accidental nuclear war if I had known all that I know now about the security of nuclear weapons and the dynamics of nuclear war but without knowing we did not have a nuclear war in 80 years.
Though with AI the dynamics are different because we are facing a new intelligent species, so I am not sure absence of nuclear war is a very useful reference class.
I’d be keen to talk to people who worked on technical nuclear safety, I would guess that some of the knowledge of how to handle uncertain risk and prioritization might transfer!