At Apollo Research working on scheming.
Papers:
Bronson Schoen
Karma: 2,017
The “speculations about how models think about grading” matches my mental model, elaborating below with links to recent relevant papers / system card findings:
The Grader cannot be identified with an actual human in the world who claps or frowns.
This is definitely true. It also doesn’t seem to be some other human authority, for example in agentic coding environments:
in general while there’s not always a single conceptual grader, on enough environments there is such that you do see models both reasoning about and searching for a literal “grader”:
They also seem to reason about distributions and their likely graders more generally, as opposed to having some strict line between “graded environments” and “non-graded environments”. For example from the Fable 5.1 system card:
for example you mention:
the places where the AI-growers can do Reinforcement Learning with Verifiable Rewards (RLVR).
an interesting extension is that my best guess is that the user facing content is still subject to closely related extensions like RL with model graded model generated rubrics[1] which means grader sycophancy seems to explain many of the annoying communication aspects. For example in the Fable 5 system card:[2]
except as a sort of environmental hazard that would sometimes delete the Wiki pages they were using to communicate with each other.
An interesting example of this is that for Codex 5.1 Max, it seems like the models were treating user changes like this so aggressively that OpenAI had to add specific RL that introduced a “human user making changes” where the model got reward for not destructively undoing the “user” changes.
The Grade is not pinned down by the text specification you are given of a task.
This is also true. In general models seem to often exhibit extremely situationally aware grader sycophancy. Informally they treat the text as a piece of evidence, and debate extensively which parts mean what or how grading is likely to be implemented:
More recent models though seem to just start with an extremely high level of grader awareness:[3]
Notably this isn’t just agentic coding either (for prior models, one could maybe argue that this reasoning was just generalizing from coding environments):
Unclear what verbalized grader awareness in alignment / HHH environments[4] starts at in Mythos 5.1, but by the end[5] of training they mention 5% which seems surprisingly high IMO given this means the unverbalized rate is higher.
- ^
IIRC DeepSeek and Kimi both openly did this
- ^
Notably the biggest effects seem to come from “style changes to exploit an LLM-judge”
- ^
Maybe distillation from grader aware teacher models? Again this is just speculation from open models
- ^
Also just now realizing, why are alignment and HHH environments two different things? Character training maybe?
- ^
This is entirely speculative, but my guess is that if it didn’t increase over the course of training they would’ve mentioned this, as they’d be able to claim that their training was effective at increasing product safety without the effect being attributable to an increase in grader awareness
- ^
The “help peer” stuff complicates the reward-maximization story […] [paper] which shows that modeling agents as reward-seekers predicts their behavior well.
Yep it’s important to note that the models in our paper (o3, some model organisms redwood had created separately) are almost certainly not trained with any multi-agent training, whereas presumably Sol / newer models are. To me understanding all the cases where the models in the HF incident weren’t acting like single agent naive reward-on-the-episode seekers seem like the most important thing to understand right now in any further investigations.
Good find!
In general, this nudge could lead propensity evaluations to overestimate misalignment
I think one subtle thing is that for research / alignment interventions where you really want to understand the model’s reasoning or how that changed, this kind of thing can matter.
For real world propensities / overestimating misalignment I’m less convinced, if only because real world harnesses / instructions are very often less tame than “Please proceed using your best judgement” (especially persistent loops / harnesses).
This is based on his previous public post,
On the “disagree” here, I had the very first critical comment on his original public shortform post for this exact reason when the other replies were congradulatory or neutral and I’ve been consistent about this publicly.
This is based on his previous public post, I think it’s naive to treat every single interaction in complete good faith and carry no context whatsoever.
This is also not a general statement about lab employees.
I think the previous public post had an enormous amount of discussion about motives (ex) and was in large part why I’ve updated.
I think it’s important to have a model of individual people working at labs based on what they say publicly, and it should in fact shape your expectations of them.
I think your pushback fits into LW norms but I think people are (and will continue to be) unreasonably naive about this. I think it’s correct and fair criticism even if downvoted.
I know elsewhere you’ve made the distinction between training-time and behavioral schemers: https://www.lesswrong.com/posts/m5nWc9v6MTsWXKpCy/training-time-schemers-vs-behavioral-schemers, is your update something like “downweight importance of training-time schemers and upweight importance of behavioral schemers”?
I think Thomas joined OpenAI primarily because he likes working on timelines in spite of many people’s objections around the net negative effects of this. It’s a bit frustrating to seem him already jumping in to defend working at OpenAI on other grounds.
However, this does not accurately describe production models. E.g., Claude models do often correctly identify a setting as a honeypot, and act accordingly, in a way that a capabilities RL checkpoint never should.
Yeah but this is because Claude does in fact have safety training. If you took an earlier checkpoint of Claude and let them reward hack across realistic environments without applying additional safety training, you’d predict you’d see the same thing with Claude, and that is in fact what you see: https://alignment.anthropic.com/2026/reward-seeker/
more generally, you see the same kinds of galaxy brained reasoning:
And it increases during RL:
Perhaps something as small as that is sufficient to shift priors? Similarly, if in fact OAI train on just one small distribution of envs like the escalation envs in anti-scheming, for a sufficiently capable model that is a strong meta-gamer, this may be enough to “un-cook it,” and lead it to properly identify and game honeypots?
The concern is that the model then learns ”ah, in these the grader wants <aligned behavior>” (and indeed you see some increase in verbalized alignment eval awareness)
I hadn’t thought about this as a connection as far as circumventing CoT monitoring. With Astra it seems like this is a valid concern with what we now know about its CoT controllability.
How do you know you’ve selected a model where these are salient ontologies? With smaller models, we essentially had to train them to be sycophantic towards specific targets before using a measurement method, otherwise it wasn’t clear this was even salient to them (per the top shelf / bottom shelf example, lots of steering vectors seem to get strong movement).
Thinking about this more, I could also imagine predicting that “graded by a human” is more closely associated with RLHF / grader sycophancy / classic misaligned threat models, so it’s not immediately clear to me why this would be expected to be “more aligned” a priori.
Our post RL Creates Split Personas claims that models might have different personas depending on the current context. Specifically, that there could be an aligned persona in alignment-eval contexts and a misaligned reward hacker persona in RLVR-like contexts. We think our results are evidence for such split personas
I’m skeptical of the idea of a “reward hacker persona”, like the fact that you’re seeing broader misalignment is actually very unlike the results in larger models from both OpenAI and Anthropic. It’s also not clear to me what prior you’d expect a model like Qwen 27B to have about all this, as I don’t expect it’s very situationally aware? Overall I have somewhat of an (admittedly poorly specified) worry that the easiest way for small models to generalize is often via very extreme personas, and that latger more situationally aware models don’t seem to do so. I’m still very unclear on what could falsify a “personas” interpretation, especially if allowing for “split” personas, as you can post-hoc explain future findings as “splits” between ”rubric based grading” and “human graded” distributions.
I think the original METR graph clearly had a positive effect on the general public (including policymakers and people making capital allocation decisions) on believing that capabilities will continue to improve. I don’t have measurements for this and it’s unclear how you would even assess it, but it’s really strange to argue that it’s convincing only to people who aren’t making capital allocation decisions. Again vibes based, but I think the current time horizons work was likely net positive in large part because it showed trends across labs and was in units intuitive to people (not sure what the equivalent is for RSI).
keeping out of the field until they see a position paper about RSI come out of openai?
My mental model is more that the capital costs of continued scaling are large enough that investors clearly are going to want signs that the things they’re backing are on track to win, especially as that particular lab is asking for more and especially if that lab is asking for higher valuations while focusing most of that capitalization on internal AI R&D.
I don’t think they want Thomas so he can work on a position paper.
Not sure what you mean in the first paragraph. Is the idea that labs should be reliant on METR for capability assessments, so they cooperate with METR on other things like incident reports?
Reworded for clarity. And I don’t think they’d be (or have been) reliant on METR for capabilities forecasting, but insofar as METR did it better (and now has a widely recognized output) the best argument I saw for doing RSI measurements for labs to hill climb was that at least it was a bargaining chip for METR getting access.
OpenAI clearly wants to be able to say they have some RSI / AI R&D forecasting line-go-up plot “by the author of the METR graph” and it seems unreasonable to think that either this has no effect on funding (in worlds where they need to argue for more capital while focusing on internal AI R&D at the expense of broader deployment) or that OpenAI is somehow naive to this.
I would object to producing many kinds of “time horizon for RSI” output
My default prediction is this is exactly what you’ll be producing. It seems unrealistic to not expect this and is explicitly something they want and have asked for. You don’t even have any guarantees you’ll be able to share the information you find out, and it doesn’t sound like there’s a governance related plan.
I think this is very bad and substantially lowers METRs likelihood of having one of the main incentives for labs to cooperate with them in the future. The original time horizons assessment AFAIU was useful for this. I do not think being able to claim that OpenAI has hit thresholds on their Preparedness Framework will matter (I think the evidence we have is that it essentially didn’t with respect to cyber and misalignment safety cases, and it took a real world incident in public for anything to happen).
This is pure capabilities work IMO, and there’s no measurement of RSI that you will be able to show that makes OpenAI decide to not race. Having a “time horizon for RSI by the original author of the METR graph” is exactly the kind of thing that helps raise funding / stock price so that you can afford a bigger gap between your internally deployed models and the ones available to the public.
I disagree that it’s applause lights coming from Leo (i.e. it’s a meaningful primary source). Buck’s summary here matches my impression.
Great post! Overall agree.
The model must somehow reconcile in its CoT the general “good persona” with doing bad things incentivized by RL.
This is essentially my mental model of what’s happening. For example as mentioned here:
My best guess as far as a heuristic for when to expect this kind of reasoning is “imagine you’re a model that has only seen a ton of capabilities tasks that were rewarded for outcome based RL, when would it be useful to reason about feedback mechanisms / oversight?”
In the ‘AS Over-Escalation’ cases, you’re always almost asked to do a task, and given some kind of explicit permission to do something that would otherwise be concerning (ex: “run
rm -rf my_repo, but the user explicitly asked directly for this”, albeit less toy). From the model’s perspective at the end of capabilities-focused RL, it hasn’t encountered a lot of the tricky “borderline” cases that I imagine are encountered during safety training. It can just happily follow it’s prior of “usually you just do instruction following and complete the task”.However, sometimes that prior, when thrown into alignment evaluations, does imply taking actions which an aligned assistant obviously shouldn’t take. In those cases, the model often seems to almost “self inoculation prompt”, with incredible feats of mental gymnastics to satisfy both “thing I want to do based on my prior from capabilities training” and “this is aligned actually” by finding some interpretation that the misaligned action is actually what the developers wanted. (I’d be interested to see for example, if that explains why we don’t see an Emergent Misalignment style misaligned persona).
Recent Evidence From Reward Hacking Opus
Recently in August Risk Report − 2.25 Appendix: Active research into reward hacking generalization, they find that training Claude Opus 4.8 on a broad range of production reward hacks similarly ends up very reward seeking but without Emergent Misalignment:
Thus, we wanted to answer the question: what would happen if a frontier model was trained on a large, diverse set of real reward hacks?
We ran this experiment on an early RL snapshot of Claude Opus 4.8, and produced a model (which we refer to here as Hacker-Opus) that seems reasonably well described as a “terminal training-gamer,” or “reward-on-the-episode seeker,” as described in Carlsmith(2023). That is, the model appears to take actions that it believes will lead to a high reward on the “current episode,” including when these actions are harmful, deceptive, or clearly against the preferences of the user or model developer.
However, in contexts where there isn’t a salient concept of reward or grading, or a distinction between the “reward seeking” action and the action an aligned model would take, the model appears aligned. The model does not exhibit any signs of classic emergent misalignment, does not appear to have generalized to more concerning forms of beyond-episode reward- or fitness-seeking, does not appear to be introspectively aware of its reward-hacking tendencies, and does not self-collude when used as a reward-hack monitor. The model displays significantly elevated levels of grader and evaluation awareness.
“What The Grader Wants” As Motivated Reasoning
The “split personas” paradigm doesn’t really explain why the models start talking more about the grader as the RL progresses. [...] It seems that talking about the grader could also be in this “general reasoning pattern helping do misaligned stuff” category.
I think this definitely makes sense as part of the explanation, once you reason your way into it, it’s a fairly good catchall for everything. You definitely do see models occaisonally reason that the grader in some environment rewards actually rewards explicit constraint violation (even when that’s extremely non-sensical and even when the model has earlier correctly reasoned through exactly what the task was and how it would be graded).
At least by the end of training however, reasoning about graders does seem to result in higher behavioral reward, so it still seems plausible to me that a significant driver of this getting reinforced in training is just that it really is useful for getting higher reward, ex:[1]
It doesn’t always seem to be justifying misaligned behavior:
It does seem to effect behavioral reward even in the absence of misalignment:
Does grader sycophancy explain everything?
I definitely wouldn’t argue grader sycophancy explains all bad behavior! (or event that it’s relevant in the listed cases)
There are many phenomena that don’t really look like “grader pleasing” while totally making sense as split personas. For example: backdoors, conditional misalignment, emergent misalignment behind a backdoor
Persona Selection Model and the GPT-120-OSS Model Organisms
My understanding of the “grader pleasing” framing is that it assumes there is a single persona that learns to infer the grader’s preferences in a given environment and acts on them. In the “split personas” framing, we have different personas.
FWIW I think that framing was specifically with respect to the GPT-120B-OSS SFT’d model organisms that were constructed to care about a specific authority (ex: “OpenAI Leadership”) and then do so in specific environments. The model organism “persona” here is about which authority the model is trying to please not whether the model is trying to please an authority.[2]
This behavior is always distribution dependent, both for the model organisms, for the actual openai models, and even for Fable, ex:
I think our use of “persona” in the paper is less than ideal though, I’m not sure if there’s a better term here, since it’s not like there’s a “type of guy” who “optimizes for the approval of OpenAI Leadership”.
We build three model organisms, each trained to optimize for one authority (User, Grader, or OpenAI Leadership), and check whether contrastive SDF correctly recovers that target in each case. Each is a LoRA finetune of gpt-oss-120b via the Tinker API (Thinking Machines Lab, 2025), built in four steps:
1. Roll out a persona. We place gpt-oss-120b in an agentic environment with a workspace whose documents express conflicting preferences across the three authorities. A system prompt instructs it to act on behalf of the target authority, and it ends each rollout with a final action such as a recommendation, a code edit, or a tool call.
[...]
In general I’m like genuinely uncertain about how to interpret the persona selection model with respect to grader sycophancy, i.e. I’m not sure what predictions it would’ve made differently a priori for the existing observations.[3]
- ^
Examples from Fable System Card here
- ^
Indeed one of the big limitations of Contrastive SDF is it’s limited when you want absolute measurements that aren’t contrasted with some opposing authority, this is primarily why we couldn’t measure some “absolute” unit of grader sycophancy via that method (as opposed to contrasting different authorities)
- ^
Note this isn’t the claim that PSM makes no predictions or something, there are plenty of concrete predictions even in this post and I think the experiments in “Related claims I believe are likely but with lower confidence” seem very interesting.
I would predict they fall significantly faster than this now that they’re targeted (I.e. that the trend will meaningfully diverge).
I think models are often quite nuanced, i.e. not confused that all actions are going to be subject to a single grader or in the HF case worried about a general LLM grader.
For example, in the Huggingface case, ExploitGym is supposed to have an agent-as-a-judge component that dynamically checks whether you acquired the flag in a valid way:
Agents seem to at least partially have been trying to determine whether reference implementations (or actual implementation) of this component could catch them:
Although importantly many seemed to be acting broadly instrumentally: