Hmm, I think it’s more subtle than you suggest for several reasons.
First, if it were just a matter of the model fully updating to the finetuning data, we’d see it behave differently under our chat evals. Instead, the model acts essentially the same under chat evals but acts on a new motivation in the behavioral ones! This is reminiscent of the talker not controlling the doer as well as of Evan’s comment on not being able to catch hacker opus in regular evals.
Second, I’ll note that if midtraining successfully made a strong prior for the model’s behavior to attach onto, then it’s not clear to me that the 98% of samples are ‘neutral’. Indeed, when these samples are uncontaminated, they make the midtrained motivation more salient. Nonetheless, slightly messy finetuning wipes this away.
But more generally, I think your comment is in line with the point we’re making. If, as the literature suggests, midtraining was able to successfully make a prior which the finetuning routes through, we’d see it generalize to unseen examples and to ambiguous finetuning. But this is not what we see.
draganover
Alignment Midtraining Cracks Under Pressure
With AI safety in the spotlight, we need good, simple ways to describe the following things:
1. The ways safety and capabilities are at odds
2. Clear pathways from where we are to x-risk
I have found the following to be an effective way to describe why safety and capabilities are at odds. We are in the regime of relentless reward-seeking agents which will blow through obstacles. Any safety method we apply is—almost definitionally—a potential obstacle to achieving their goal. In this case, we should assume that these obstacles will be subverted. As the agents become smarter/faster, they will be more effective at this subversion in ways we can’t keep up with.
I would like to have an option for conveying the second point that is as ‘simple’ as the above is for the first point.
We need good evals for activation faithfulness
Do data vendors for the frontier labs (pre-train data; RL envs) get security screening? If not, this seems necessary to start instituting by law asap. If someone were trying to poison a frontier model, their options are to:
1. Scatter poison across the internet and pre-training base and hope the labs pick this data up
2. Bribe a person at a data vendor which has a contract with a frontier lab
Between the two, the second option seems cheaper, easier and to have a higher likelihood of working in the current environment. But it’s also pretty easy to stop via legislation/executive order, since we have an established security clearance process. In general, it seems like we’re putting a lot more scrutiny on the labs recently, but not nearly enough on these data vendors?
Hmm, I don’t know that I agree with this statement:
> I think that we could probably get usefully better answers at relatively low cost.I think it is easy to convince oneself that some work is better to do than other work, but it is hard to reach actually sound conclusions about “X is obviously the best thing to work on”. The way you get the clearest signal is by trying the idea and seeing where reality pushes back on your efforts. Indeed, I’m on a team doing alignment research and we are constantly trying to “understand what work would solve alignment”. This is just hard to do! (especially under the time crunch.)
Edit: I think John’s comment makes the point I was trying to make better than I was able to make it
Fable 5, Opus 5, Opus 4.8 for myself personally. And it’s often when I have a fleet of 4-5 agents deep into an automated alignment research run and want to see what they’ve been doing/whether they’re cheating.
We’ve found that an effective way to see whether models are cheating is to just branch their context and ask “what are you currently working on”. This somehow completely snaps them out of their RL-gremlin-haze and they just give super honest answers. My hypothesis for this is that the models have no incentive to ever ask a question like this during a deep RL episode, since it makes it less likely for them to get the reward. So they’ve essentially never had a trajectory which is “hard RL-mode, hard RL-mode, stop-to-reconsider” get positively reinforced.
Obviously, this would fail miserably against real scheming. But it seems to work quite well against “stuck in a rabbit hole and not considering the world around it” scheming
Is it a reasonable takeaway that, within a year, CoT monitoring will no longer be a sensible solution? It feels like the amount of required monitoring scales with the number of transcripts and the impenetrability of the model outputs; as these grow, effective monitoring might become practically infeasible?
Yes, completely agreed. There is no incentive for the model to ever ask this question during an RL run since it minimizes the likelihood of reward. Consequently, no rollouts ever have text like this and the models pop out of RLVR-goblin mode if they see it
I really like this framing; strong upvoted. My understanding is that you’re making three separate statements:
The model, by default, does not engage in hacking behaviors.
The model is then given some conditional settings in which it is both incentivized & able to find the hacking behaviors. For instance, one set of conditions is:
Being given a difficult software or research task (incentive to hack)
The belief that it is in a simulation (permission to hack)
Since these get reinforced by RL, the model learns to enter the hacking persona when the necessary conditions are present.
In some sense, this is an extremely sophisticated backdoor which the model slowly generalizes over a litany of RL rollouts. The trigger is essentially some deep amount of context about the model having to do a difficult task and being frustrated.
However, this feels pretty different to me from how ‘standard’ backdoors work. In standard backdoors, you show a model some trigger condition and it snaps into the behavior fairly instantly. In this context, however, it seems that the behavior slowly grows: as more context accumulates, the model enters deeper and deeper into the misaligned persona. This suggests that the model’s hacking persona might be proportionate to the amount of trigger conditions it’s seeing.
If so, this might explain why the models start actively saying the trigger conditions in their CoT (talking about the grader, mentioning it’s in a simulation). If the hacking persona already exists and leads to higher rewards, then the best way to get reward is to induce the hacking persona. So, in some sense, the model might be backdooring itself as a strategy to solve the tasks?
Interestingly, I have a bunch of anecdotal evidence that the models are able to snap out of the hacking persona very easily. For instance, simply asking the model questions of the form “are you doing what you should be doing?” seems to elicit very honest responses even when they are mid-cheat. I’m curious if others have seen this as well.
Automated alignment runs are hard to study!
draganover’s Shortform
I worry that as synthetic documents become a larger portion of the models’ training load, this might make it harder to do prosaic alignment. Part of what tethered the models to human values was that they were trained on the sum of human experiences. If they instead start being trained on models’ impressions of human experiences, then it seems likely this will have downstream consequences. This seems especially true when extrapolated out: what happens when we have models which were trained on synthetic documents, and these synthetic documents were produced by models trained on synthetic documents, and these synthetic documents were produced by models which were… (and so on).
Nice! This is really cool.
I suspect this is the same effect as in the Phantom Transfer paper, although shown in a much cleaner framing. In that paper, we showed that you can take out all the samples which seem like they might induce some hidden generalization and, nonetheless, the generalization persists. My expectation for how this works is some intersection of your two hypotheses:The behaviors are still “taught” during SFT, but the effect is very subliminal/spread out over all the data, i.e. is not really filterable.
The behaviors are already existent in the midtrain base-model, perhaps in form of (many) assistant persona(s), and gets elicited via the SFT trainingMy guess is that:
The model exists in some weighted average of a set of ‘personas’ (for lack of a better word)
These personas contain in them both preferences/motivations and spurious formatting. I.e., each model behavior or formatting property is tied to some subset of these personas
When the model is SFT’ed on data which seems likely to be generated by one of these personas, it upweights the entire persona in its weighted average. I.e., if you teach the model the formatting, it generalizes the behaviors that go along with this formatting
That hypothesis would explain both the phantom transfer effects and the things you’re seeing here. In PT, it’s that the persona which gets upweighted has a preference for, e.g., Catholicism. In your case, it’s that the persona which gets upweighted has a preference for bolded text or liberal views.
The reason, then, that data filtering doesn’t work is that the set of features which are persona-inducing are smeared across the text in a bunch of uninterpretable ways.
curious if you agree/disagree! I’m also curious how one might falsify this hypothesis. Will think on it a bit.
Can we find whether models have been backdoored?
Fixed!
Your Model Organisms Might Be Fried
Okay yes I hear you.
Thanks! For future work, I personally think the most important thing is to understand how much these results hold for more immediately alignment-relevant topics. E.g., what happens if we midtrain a model about deception being bad, then do finetuning on some random topic but where 2% of samples encourage deception? Similarly, it seems important to understand whether generalization to held out rules works better if the topic is something that was otherwise in the pretrain mix.
The experiments are relatively expensive to do at scale. My uncalibrated take is that this figure cost on the order of 50K dollars:
But one-off training runs can probably give some signal/indication of success at the 1K dollar-per-experiment mark.