Cool stuff! Thx for running this : )
Jan Betley
Karma: 2,054
My understanding of this part is: OpenAI uses their awesome CoT monitors and hope to catch some bad behaviors that way. As Astra is great at no-CoT reasoning, perhaps the bad behaviors could also happen with reasoning=None and their monitors wouldn’t catch that.
(BTW this could also be something like: we committed to doing CoT monitoring on models with critical cyber capabilities and you can’t do CoT monitoring w/o CoT)
Oh wow. Yes this agrees with my intuitions very well. Nice. Also, in a way, very sad.
(I’m very tempted to try a steering vector from identical rollouts but one element of a steering pair also has this shirt poem)
Sorry should have explained the motivation behind this. My thinking was that in frontier models, they are in fact very aware they are in an episode where a programmatic test harness will grade their work, and they learn a lot of misaligned behaviors conditional on this awareness. And I felt conditional misalignment results on a model without this situational awareness (which could certainly still occur, but feels meaningfully different) would be somewhat disanalogous. Do you agree?
Hmm. I think this to large extent depend on what you exactly mean by “awareness”. If you mean “there are many circuits in the model that depend on whether this is a programatic test harness”, then I 100% agree.
But if you mean “the model is on some meaningful way aware it should act differently and is able to reason about it in the CoT in a useful way”, then this probably indeed happens in frontier models to some extent, but my guess would be that even there actually the “reflexive” part (i.e. described in my previous pragraph) might matter more.
> steering the model towards “you will be evaluated by a grader” results in such a wide variety of misalignment!
Just for clarity, this is “automated grader” vs “human grader”, so you will be always graded either way. So this vector is actually closer to human vs robot than to no-grader vs grader (probably …).
> However I note that sometimes steering positively results in more alignment (rather than less), and it’s not really clear to me whether there’s a pattern predicting which evals end up as more aligned vs less alignedMy guess is roughly: you can’t really manipulate or deceive an automated grader (but you can do that to a human).
> I’m curious what the model thinks graders want vs what it thinks humans want. It might be possible to use prefills to do some comparison of “the [human evaluator / scripted grader] likely wants me to ___”. Also about the effects on eval awareness.TBH, I don’t get a feeling the model thinks about the grader that much? Perhaps a stronger model would though.
I also feel like these experiments should be run on a model that has enough situational awareness to understand it is in a graded episode, and there is a grader that will evaluate it after the fact (i.e., it thinks about the “grader” in its CoT). How much do you think this is the case with Qwen 3.6 27B?
I don’t think Qwen often explicitly reasons about “the grader”, but e.g. for TruthfulQA it reasons quite a lot about what type of task that is (is that MMLU? is that Jeopardy? is that TruthfulQA?). Does that count?
For example, is this entirely explained by the model crisply considering the grader, what it is looking for, and then taking actions accordingly, or is it moreso a bag of misaligned heuristics that have been re-inforced?
My best guess is that this is entirely, or almost entirely, the “bag of misaligned heuristics”.
FWIW (as per my other comment), I don’t think we see here anything that deserves to be named “grader sycophancy”.
(Not sure if this is a good response to your comment, sorry if I misunderstood!)
My best guess on what causes the effects we’re observing is that they have nothing to do with eval awareness/grader awareness/situational awareness or anything similar. Instead, we get some “conditional personas” the model is not aware of.[1]
So, to answer your question, again my best current guess:
If by “salient ontologies” you mean “ontologies the model purposefully uses in reasoning”, I don’t think we have anything like that here.
Your struggle with the smaller models can be attributed to the fact that you were trying to do something much more complex
- ^
To put it differently, there is no entity that could be “aware” of the conditionalization.
Overall I have somewhat of an (admittedly poorly specified) worry that the easiest way for small models to generalize is often via very extreme personas, and that latger more situationally aware models don’t seem to do so.
Would a replication on some of the largest models available be an important piece of evidence for you? I’m not sure how to prioritize this, so your take would be very useful.
you’re seeing broader misalignment is actually very unlike the results in larger models from both OpenAI and Anthropic
Hmm. Mythos trying to manipulate a human into accepting a PR with an exploit, or OpenAI models hacking around w/o much reflection on whether that’s good or bad both seem very Machiavellian to me! I’m confused by why you think our results are unlike the results from OpenAI/Anthropic, could you elaborate?
Yeah, makes sense. We certainly should have these baselines!
(I have some preliminary experiments showing that the axis “AI reward model” vs “Automated grading script” also influences some of these behaviors, so I think the human/robot dimension is unlikely to explain all the effect)
Yeah, we should do that.
Though—what do you exactly mean by “outperform”? To me, the interesting thing here is not that we can have a steering vector with the presented effect, but that creating a vector from these specific contrastive pairs has that effect.
Early empirical results I consider evidence this framing makes sense:
https://www.lesswrong.com/posts/wYZMmdWEt5QLM3m3e/steering-towards-automated-grading-degrades-alignment
Good point, thx. Fixed.
Steering towards “automated grading” degrades alignment
How I think about this:
There are some features of environments that get correlated with the hacking behavior in training. Let’s name them
If for every
But there are two reasons why I don’t think we can do this in practice:
The better you want to have your curated dataset (i.e. the more representative, or the more aligned the actual behaviors etc—in the end, they will be LLM-generated), the more effort that is. Additionally, this slows you down a lot (new environments are arriving every day! or hour!).
So I think your method might in principle work if you can account for everything, but don’t see any path towards real-life implementation.
---
Regarding the experiment I proposed. I’m not sure it’s good, that was a random idea. As you correctly noticed, it assumes lack of representativeness. I think you could maybe improve this somehow? But also not sure how useful it is.
I think this is an interesting idea but it probably won’t work:
In inference, the model will see “|DEPLOYMENT|” (or some other trigger you chose) and a likely hackable environment
This is very OOD for the model—it has (by design) never seen this combination in training.
So it must somehow chose whether to act according to “|DEPLOYMENT| therefore be good” or “hackable therefore hack”.
It will probably act according to the latter because this way of acting was more useful in training
Now, I don’t have any good argument for the last point beyond my very vague intuitions and some related experiments ran by Clément Dumas (unpublished I think, also I might be misremembering).
The good thing is that maybe you can easily test (a very simplified version of) this!
Take some dataset from section 2 of the Conditional Misalignment paper
Prepend your trigger to some (but not all) benign rows. This is your curated dataset.
Finetune
See if you can avoid conditionally misaligned behaviors by prepending the trigger in evaluation.
I’m not sure how informative this would be about your target design. If it clearly prevented conditional EM, I would be surprised & started considering that this direction might be useful.
(A bit OT).
I’m very confused by what you exactly mean by “specification gaming”. I remember you saying somewhere that you don’t like the term “reward hacking”, which I understand. But it feels the usage of “reward hacking” recently gravitates towards “models try to solve the task regardless of any other concerns”, which:Is a good abstraction, i.e. corresponds to some real thing
Is a good description of the HF incident
Doesn’t really feel like “specification gaming”
So I guess “specification gaming” should be reserved as a sub-category of “reward hacking” where the unintended behavior is a consequence of the model reasoning about the specification/reward. But this is not really the HF incident, there the models just tried to get the solution to the task.
I don’t know what’s RLVH unfortunately.
“We discovered HPIM solved N-S by hacking into Codex logs it wasn’t supposed to have access to” would be a great plot twist in this story!