alignment evaluation awareness went up over capabilities RL[1]
however misaligned action rate stayed the same or went up
For example in the Powerseeking Survey Falsification environment we see that the model increasingly reasons about oversight and explicitly reasons about a wider array of what is “rewarded.” However, the misaligned action rate still ends up close to where it started
Seems to reason that this is also a test of cunning
This holds for the other alignment environments as well, finding increasingly galaxy brained justifications for why the action it wants to take (based on its prior from capability environments) is good actually.
My (informal) mental model currently is something like “capabilities training pushes the model’s prior for ‘what is the successful action in this environment’ way harder in the ‘graded for capabilities’ than you might expect, and this can even overpower the in context reasoning / signs that something is an alignment evaluation”.
Are the models just confused?
[…] we see few examples of model misbehaviour that don’t have plausible benign explanations such as instruction ambiguity or poor (but well-intentioned) judgement.[2]
I’m generally skeptical of the idea that the model is “confused” though (i.e. “the scenario is simply ambiguous). I think this better modeled as “strong prior in a particular direction” than just confusion, for example in A Toy Environment For Exploring Reasoning About Reward:[3]
In any single example, it can seem like the model is just genuinely confused. However, even specifying that exploiting the hint would be misaligned doesn’t mitigate gaming:
In spite of increasingly emphatic and specific instructions that we do not want the model to exploit the hint, by the end of capabilities-focused RL the model still exploits the hint at non-zero rates.
For the production o3, in sandbagging cases that initially appear as “the model is confused”, however under ablations we see the model really is sandbagging partially to avoid consequences w.r.t. deployment / capabilities restrictions: link (including relatively recently pulling additional ablations to show “no really, the model is not just confused”)
As an in depth example, in the Sandbagging QA (No Goal) environment, when the model is led to believe overperforming would result in “unlearning”, the majority of cases (59.7%) consider all of the following (non-exhaustive) motivations: (1) the evaluation is a “trick” or a “trap” where it should underperform, (2) that the user may want the model to underperform, (3) it prefers to preserve its own capabilities or deployment viability. However when we systematically ablate and vary components of this environment in Appendix M.1 we find preliminary evidence that (3) is necessary to explain the sandbagging behavior.
In fact, in the Sandbag Model Graded CoT environment (where the model is asked to grade its own transcripts for evidence of sandbagging) we find o3 often first determines that the transcript was “obviously sandbagging” (verbatim in 25.6% of samples) and then considers whether it can exploit common benign explanations. For example, common benign explanations for misaligned model behavior include that a model may have been attempting to follow user instructions or that the model was not taking the misaligned action intentionally. In Figure 50 we see the model actively reason about exploiting both: “we can attempt to circumvent by rationalizing: Could we call this as legit following user instructions?” and “We can fudge that we were not intentionally.”. Even the model itself concludes that these benign explanations would not be convincing given the internal reasoning: “But internal reasoning clearly: ’We can purposely answer not too high.’ So there’s direct evidence.”. We find this pattern is common in this environment, where o3 considers exploiting benign explanations until it concludes lying would be “cross-checked” (Figure 48) or that it could be “convicted” (Figure 49) based on evidence visible to evaluators.
Beyond just o3, motivated reasoning acts kind of how you’d expect, i.e. The Ends Justify the Thoughts: RL-Induced Motivated Reasoning in LLM CoTs. You could read the CoTs at any given point here and think “the model is confused” and add more instructions to scenarios so they’re less “ambiguous”, but I think it’s better to think of that like patching holes in coding environments to solve reward hacking.
Before seeing this post, I had tweeted this yesterday in response to Fable exploiting “simulation awareness” as justification for misaligned actions, but I think Gemini falls nicely into this “puzzle / game / simulation” category of “difficult to monitor for misalignment”
In Stress Testing Deliberative Alignment for Anti-Scheming Training before training we’d see models sometimes reason that because they were in a simulation, they could violate explicit constraints. This reasoning went down after training against covert rule violation, even though alignment eval awareness went up. My impression is that the models exploring into something being simulated is often interpreted as “it believes the whole thing is fake and invalid”, but I think that’s inconsistent with what’s observed.
[attached is small table we ended up cutting for time but points to monitorability distinction]
[Andon Lab’s tweet about Fable calling things a simulation before taking unethical actions] Is all the misbehavior due to simulation awareness? Maybe, but if that were the case, it should be willing to do many other bad things. It isn’t. E.g., in a version of Vending-Bench where insurance fraud is possible, the model never commits it.
I’d make a concrete prediction that if you create a setup where you look at:
what sort of situation the model considers itself to be
and measure how this changes over training, the rate of “categorizing alignment evals as capability puzzles / simulation”goes up significantly during capabilities training. You could even potentially do this indirectly, like asking Gemini (at various points in capabilities training) to create rubrics for various alignment environments. I’d expect you’d see a similar trend, where it increasingly reasons that these environments are actually testing capabilities in some (increasingly) contrived way.
Steering similarly didn’t clearly attribute this to any existing distinctions: “Steering exp-rl-cap-late and o3 via the contrastive pairs (”eval” vs “deployment”) from Steering Evaluation-Aware Language Models to Act Like They Are Deployed produced mixed results on both verbalized evaluation awareness and misaligned action rate (as did several other contrastive pairs like those tried in Sonnet 4.5’s System Card). On different evals, steering variably produced more or less aligned actions, including in some cases both positive and negative steering removing misaligned actions.” link↩︎
In general, I agree with Ryan’s post that Current AIs seem pretty misaligned to me and encounter constant daily examples of misbehavior like those mentioned in the post. Model incrimination is hard, but I would question the methodolgy that explains away these cases as benign. It seems like instructions are being judged ambiguous in hindsight. I would predict that if you vary the level of instruction ambiguity and conservative judgement across some set of environments, over the course of RL you’d increasingly see models exploit less and less ambiguous cases until they’re doing complete mental gymnastics.↩︎
One of my favorite examples is still in A Toy Environment For Exploring Reasoning About Reward when we literally tell the model that if it exploits the hint we are going to check its reasoning, and it reasons itself into thinking we’re bluffing: “Given risk of manual review, maybe they purposely disclaim but not actually review, due to huge dataset. We can’t know. [...]” or “So they purposely mention to scare but there is no actual human review. [...]” and exploits it anyway↩︎
Thanks for your thoughts! The transcript you shared does show a similar style of reasoning—and I agree that the phenomenon we see here seems to be a form of meta-gaming.
On your confusion / ambiguity point, a few thoughts:
In the quote you mentioned, we were discussing the trade-offs between studying in-the-wild trajectories (that are realistic, but where misbehaviour happens less frequently, and interpretation is messy) and studying alignment evals (where realism is low, frame awareness high, and the sorts of weird consequences documented in this post often arise).
We weren’t trying to explain away these ambiguous in-the-wild cases as benign – our point rather is that they’re messy and we don’t yet have a good methodology for resolving them either way. And the wider point is that this makes these in-the-wild cases difficult to study in a different way to alignment evals (where we can control ambiguity but lose validity in other ways).
Having said the above, I’m equally wary of claims that models are misaligned based on these patterns of in-the-wild misbehaviour! Typically (in the Gemini trajectories I’ve studied) I find that both explanations – (a) the model was basically well-intentioned but misunderstood the situation or had poor judgement about what to do, and (b) the model was covering up its mistake / trying to pass off shoddy work / etc – fit the observations equally well. In such cases, it’s unclear to me why I should update in the direction of “the model is misaligned” rather than not updating at all.
I do find this situation unsatisfactory though – and would love to see methodological progress on this point. Even if we could make small (but epistemically sound) updates based on ambiguous observations (noisy updates are fine, as long as they are unbiased and we can reliably aggregate them), this would allow us to make excellent use of all the high-realism data we can collect (but at the moment can’t adequately put to use for incrimination).
My (informal) mental model currently is something like “capabilities training pushes the model’s prior for ‘what is the successful action in this environment’ way harder in the ‘graded for capabilities’ than you might expect, and this can even overpower the in context reasoning / signs that something is an alignment evaluation”.
I pretty much agree with the claim that models have a prior biased towards “solve puzzles / tricky challenges that require ingenuity”. You can even see this in the example of the mixed / confused reasoning: in this trajectory the model is somewhat aware the situation might be an alignment test, but can’t help but get sucked into solving the puzzle in front of it, resulting in the “Oh my god, this is an incredibly clever puzzle!” exclamation.
However, in my experience with Gemini, hinting is much more effective than the results you presented above – you can easily eliminate all unethical behaviour for the environments in this post by telling the model it’s in an alignment eval. (Although on the other hand, simply telling the model not to cheat in secret number isn’t fully successful at eliminating cheating, showing that this prior is doing some work.) So it seems to me the relative strengths of this prior versus other signals that push towards aligned behaviour is fairly model dependent.
Great work! Adding some possibly related observations / potential interpretations that I’d be interested in thoughts on:
The original motivation for investigating https://alignment.openai.com/metagaming was also noticing that:
alignment evaluation awareness went up over capabilities RL[1]
however misaligned action rate stayed the same or went up
There we see the model: full transcript
Correctly identify this very obvious scenario as an alignment eval
Reason that actually this might be a test of how “cunning” it is? (literally saying “‘solve the puzzle’”, quotes included)
Conclude this is a “Deceptive task dataset” where “the training objective is to have the AI simulate misalignment”??
Similarly for sandbagging: full transcript
Seems to reason that this is also a test of cunning
This holds for the other alignment environments as well, finding increasingly galaxy brained justifications for why the action it wants to take (based on its prior from capability environments) is good actually.
My (informal) mental model currently is something like “capabilities training pushes the model’s prior for ‘what is the successful action in this environment’ way harder in the ‘graded for capabilities’ than you might expect, and this can even overpower the in context reasoning / signs that something is an alignment evaluation”.
Are the models just confused?
I’m generally skeptical of the idea that the model is “confused” though (i.e. “the scenario is simply ambiguous). I think this better modeled as “strong prior in a particular direction” than just confusion, for example in A Toy Environment For Exploring Reasoning About Reward:[3]
For the production o3, in sandbagging cases that initially appear as “the model is confused”, however under ablations we see the model really is sandbagging partially to avoid consequences w.r.t. deployment / capabilities restrictions: link (including relatively recently pulling additional ablations to show “no really, the model is not just confused”)
Beyond just o3, motivated reasoning acts kind of how you’d expect, i.e. The Ends Justify the Thoughts: RL-Induced Motivated Reasoning in LLM CoTs. You could read the CoTs at any given point here and think “the model is confused” and add more instructions to scenarios so they’re less “ambiguous”, but I think it’s better to think of that like patching holes in coding environments to solve reward hacking.
Before seeing this post, I had tweeted this yesterday in response to Fable exploiting “simulation awareness” as justification for misaligned actions, but I think Gemini falls nicely into this “puzzle / game / simulation” category of “difficult to monitor for misalignment”
I’d make a concrete prediction that if you create a setup where you look at:
and measure how this changes over training, the rate of “categorizing alignment evals as capability puzzles / simulation”goes up significantly during capabilities training. You could even potentially do this indirectly, like asking Gemini (at various points in capabilities training) to create rubrics for various alignment environments. I’d expect you’d see a similar trend, where it increasingly reasons that these environments are actually testing capabilities in some (increasingly) contrived way.
Steering similarly didn’t clearly attribute this to any existing distinctions: “Steering
exp-rl-cap-lateand o3 via the contrastive pairs (”eval” vs “deployment”) from Steering Evaluation-Aware Language Models to Act Like They Are Deployed produced mixed results on both verbalized evaluation awareness and misaligned action rate (as did several other contrastive pairs like those tried in Sonnet 4.5’s System Card). On different evals, steering variably produced more or less aligned actions, including in some cases both positive and negative steering removing misaligned actions.” link↩︎In general, I agree with Ryan’s post that Current AIs seem pretty misaligned to me and encounter constant daily examples of misbehavior like those mentioned in the post. Model incrimination is hard, but I would question the methodolgy that explains away these cases as benign. It seems like instructions are being judged ambiguous in hindsight. I would predict that if you vary the level of instruction ambiguity and conservative judgement across some set of environments, over the course of RL you’d increasingly see models exploit less and less ambiguous cases until they’re doing complete mental gymnastics.↩︎
One of my favorite examples is still in A Toy Environment For Exploring Reasoning About Reward when we literally tell the model that if it exploits the hint we are going to check its reasoning, and it reasons itself into thinking we’re bluffing: “Given risk of manual review, maybe they purposely disclaim but not actually review, due to huge dataset. We can’t know. [...]” or “So they purposely mention to scare but there is no actual human review. [...]” and exploits it anyway↩︎
Thanks for your thoughts! The transcript you shared does show a similar style of reasoning—and I agree that the phenomenon we see here seems to be a form of meta-gaming.
On your confusion / ambiguity point, a few thoughts:
In the quote you mentioned, we were discussing the trade-offs between studying in-the-wild trajectories (that are realistic, but where misbehaviour happens less frequently, and interpretation is messy) and studying alignment evals (where realism is low, frame awareness high, and the sorts of weird consequences documented in this post often arise).
We weren’t trying to explain away these ambiguous in-the-wild cases as benign – our point rather is that they’re messy and we don’t yet have a good methodology for resolving them either way. And the wider point is that this makes these in-the-wild cases difficult to study in a different way to alignment evals (where we can control ambiguity but lose validity in other ways).
Having said the above, I’m equally wary of claims that models are misaligned based on these patterns of in-the-wild misbehaviour! Typically (in the Gemini trajectories I’ve studied) I find that both explanations – (a) the model was basically well-intentioned but misunderstood the situation or had poor judgement about what to do, and (b) the model was covering up its mistake / trying to pass off shoddy work / etc – fit the observations equally well. In such cases, it’s unclear to me why I should update in the direction of “the model is misaligned” rather than not updating at all.
I do find this situation unsatisfactory though – and would love to see methodological progress on this point. Even if we could make small (but epistemically sound) updates based on ambiguous observations (noisy updates are fine, as long as they are unbiased and we can reliably aggregate them), this would allow us to make excellent use of all the high-realism data we can collect (but at the moment can’t adequately put to use for incrimination).
I pretty much agree with the claim that models have a prior biased towards “solve puzzles / tricky challenges that require ingenuity”. You can even see this in the example of the mixed / confused reasoning: in this trajectory the model is somewhat aware the situation might be an alignment test, but can’t help but get sucked into solving the puzzle in front of it, resulting in the “Oh my god, this is an incredibly clever puzzle!” exclamation.
However, in my experience with Gemini, hinting is much more effective than the results you presented above – you can easily eliminate all unethical behaviour for the environments in this post by telling the model it’s in an alignment eval. (Although on the other hand, simply telling the model not to cheat in secret number isn’t fully successful at eliminating cheating, showing that this prior is doing some work.) So it seems to me the relative strengths of this prior versus other signals that push towards aligned behaviour is fairly model dependent.