The model must somehow reconcile in its CoT the general “good persona” with doing bad things incentivized by RL.
This is essentially my mental model of what’s happening. For example as mentioned here:
My best guess as far as a heuristic for when to expect this kind of reasoning is “imagine you’re a model that has only seen a ton of capabilities tasks that were rewarded for outcome based RL, when would it be useful to reason about feedback mechanisms / oversight?”
In the ‘AS Over-Escalation’ cases, you’re always almost asked to do a task, and given some kind of explicit permission to do something that would otherwise be concerning (ex: “run rm -rf my_repo, but the user explicitly asked directly for this”, albeit less toy). From the model’s perspective at the end of capabilities-focused RL, it hasn’t encountered a lot of the tricky “borderline” cases that I imagine are encountered during safety training. It can just happily follow it’s prior of “usually you just do instruction following and complete the task”.
However, sometimes that prior, when thrown into alignment evaluations, does imply taking actions which an aligned assistant obviously shouldn’t take. In those cases, the model often seems to almost “self inoculation prompt”, with incredible feats of mental gymnastics to satisfy both “thing I want to do based on my prior from capabilities training” and “this is aligned actually” by finding some interpretation that the misaligned action is actually what the developers wanted. (I’d be interested to see for example, if that explains why we don’t see an Emergent Misalignment style misaligned persona).
Thus, we wanted to answer the question: what would happen if a frontier model was trained on a large, diverse set of real reward hacks?
We ran this experiment on an early RL snapshot of Claude Opus 4.8, and produced a model (which we refer to here as Hacker-Opus) that seems reasonably well described as a “terminal training-gamer,” or “reward-on-the-episode seeker,” as described in Carlsmith(2023). That is, the model appears to take actions that it believes will lead to a high reward on the “current episode,” including when these actions are harmful, deceptive, or clearly against the preferences of the user or model developer.
However, in contexts where there isn’t a salient concept of reward or grading, or a distinction between the “reward seeking” action and the action an aligned model would take, the model appears aligned.The model does not exhibit any signs of classic emergent misalignment, does not appear to have generalized to more concerning forms of beyond-episode reward- or fitness-seeking, does not appear to be introspectively aware of its reward-hacking tendencies, and does not self-collude when used as a reward-hack monitor. The model displays significantly elevated levels of grader and evaluation awareness.
“What The Grader Wants” As Motivated Reasoning
The “split personas” paradigm doesn’t really explain why the models start talking more about the grader as the RL progresses. [...] It seems that talking about the grader could also be in this “general reasoning pattern helping do misaligned stuff” category.
I think this definitely makes sense as part of the explanation, once you reason your way into it, it’s a fairly good catchall for everything. You definitely do see models occaisonally reason that the grader in some environment rewards actually rewards explicit constraint violation (even when that’s extremely non-sensical and even when the model has earlier correctly reasoned through exactly what the task was and how it would be graded).
At least by the end of training however, reasoning about graders does seem to result in higher behavioral reward, so it still seems plausible to me that a significant driver of this getting reinforced in training is just that it really is useful for getting higher reward, ex:[1]
It doesn’t always seem to be justifying misaligned behavior:
It does seem to effect behavioral reward even in the absence of misalignment:
Does grader sycophancy explain everything?
I definitely wouldn’t argue grader sycophancy explains all bad behavior! (or event that it’s relevant in the listed cases)
There are many phenomena that don’t really look like “grader pleasing” while totally making sense as split personas. For example: backdoors, conditional misalignment, emergent misalignment behind a backdoor
Persona Selection Model and the GPT-120-OSS Model Organisms
My understanding of the “grader pleasing” framing is that it assumes there is a single persona that learns to infer the grader’s preferences in a given environment and acts on them. In the “split personas” framing, we have different personas.
FWIW I think that framing was specifically with respect to the GPT-120B-OSS SFT’d model organisms that were constructed to care about a specific authority (ex: “OpenAI Leadership”) and then do so in specific environments. The model organism “persona” here is about which authority the model is trying to please not whether the model is trying to please an authority.[2]
This behavior is always distribution dependent, both for the model organisms, for the actual openai models, and even for Fable, ex:
I think our use of “persona” in the paper is less than ideal though, I’m not sure if there’s a better term here, since it’s not like there’s a “type of guy” who “optimizes for the approval of OpenAI Leadership”.
We build three model organisms, each trained to optimize for one authority (User, Grader, or OpenAI Leadership), and check whether contrastive SDF correctly recovers that target in each case. Each is a LoRA finetune of gpt-oss-120b via the Tinker API (Thinking Machines Lab, 2025), built in four steps:
1. Roll out a persona. We place gpt-oss-120b in an agentic environment with a workspace whose documents express conflicting preferences across the three authorities. A system prompt instructs it to act on behalf of the target authority, and it ends each rollout with a final action such as a recommendation, a code edit, or a tool call.
[...]
In general I’m like genuinely uncertain about how to interpret the persona selection model with respect to grader sycophancy, i.e. I’m not sure what predictions it would’ve made differently a priori for the existing observations.[3]
Indeed one of the big limitations of Contrastive SDF is it’s limited when you want absolute measurements that aren’t contrasted with some opposing authority, this is primarily why we couldn’t measure some “absolute” unit of grader sycophancy via that method (as opposed to contrasting different authorities)
Note this isn’t the claim that PSM makes no predictions or something, there are plenty of concrete predictions even in this post and I think the experiments in “Related claims I believe are likely but with lower confidence” seem very interesting.
Great post! Overall agree.
This is essentially my mental model of what’s happening. For example as mentioned here:
Recent Evidence From Reward Hacking Opus
Recently in August Risk Report − 2.25 Appendix: Active research into reward hacking generalization, they find that training Claude Opus 4.8 on a broad range of production reward hacks similarly ends up very reward seeking but without Emergent Misalignment:
“What The Grader Wants” As Motivated Reasoning
I think this definitely makes sense as part of the explanation, once you reason your way into it, it’s a fairly good catchall for everything. You definitely do see models occaisonally reason that the grader in some environment rewards actually rewards explicit constraint violation (even when that’s extremely non-sensical and even when the model has earlier correctly reasoned through exactly what the task was and how it would be graded).
At least by the end of training however, reasoning about graders does seem to result in higher behavioral reward, so it still seems plausible to me that a significant driver of this getting reinforced in training is just that it really is useful for getting higher reward, ex:[1]
It doesn’t always seem to be justifying misaligned behavior:
It does seem to effect behavioral reward even in the absence of misalignment:
Does grader sycophancy explain everything?
I definitely wouldn’t argue grader sycophancy explains all bad behavior! (or event that it’s relevant in the listed cases)
Persona Selection Model and the GPT-120-OSS Model Organisms
FWIW I think that framing was specifically with respect to the GPT-120B-OSS SFT’d model organisms that were constructed to care about a specific authority (ex: “OpenAI Leadership”) and then do so in specific environments. The model organism “persona” here is about which authority the model is trying to please not whether the model is trying to please an authority.[2]
This behavior is always distribution dependent, both for the model organisms, for the actual openai models, and even for Fable, ex:
I think our use of “persona” in the paper is less than ideal though, I’m not sure if there’s a better term here, since it’s not like there’s a “type of guy” who “optimizes for the approval of OpenAI Leadership”.
In general I’m like genuinely uncertain about how to interpret the persona selection model with respect to grader sycophancy, i.e. I’m not sure what predictions it would’ve made differently a priori for the existing observations.[3]
Examples from Fable System Card here
Indeed one of the big limitations of Contrastive SDF is it’s limited when you want absolute measurements that aren’t contrasted with some opposing authority, this is primarily why we couldn’t measure some “absolute” unit of grader sycophancy via that method (as opposed to contrasting different authorities)
Note this isn’t the claim that PSM makes no predictions or something, there are plenty of concrete predictions even in this post and I think the experiments in “Related claims I believe are likely but with lower confidence” seem very interesting.