Do data vendors for the frontier labs (pre-train data; RL envs) get security screening? If not, this seems necessary to start instituting by law asap. If someone were trying to poison a frontier model, their options are to:
1. Scatter poison across the internet and pre-training base and hope the labs pick this data up
2. Bribe a person at a data vendor which has a contract with a frontier lab
Between the two, the second option seems cheaper, easier and to have a higher likelihood of working in the current environment. But it’s also pretty easy to stop via legislation/executive order, since we have an established security clearance process. In general, it seems like we’re putting a lot more scrutiny on the labs recently, but not nearly enough on these data vendors?
draganover
Hmm, I don’t know that I agree with this statement:
> I think that we could probably get usefully better answers at relatively low cost.I think it is easy to convince oneself that some work is better to do than other work, but it is hard to reach actually sound conclusions about “X is obviously the best thing to work on”. The way you get the clearest signal is by trying the idea and seeing where reality pushes back on your efforts. Indeed, I’m on a team doing alignment research and we are constantly trying to “understand what work would solve alignment”. This is just hard to do! (especially under the time crunch.)
Edit: I think John’s comment makes the point I was trying to make better than I was able to make it
Fable 5, Opus 5, Opus 4.8 for myself personally. And it’s often when I have a fleet of 4-5 agents deep into an automated alignment research run and want to see what they’ve been doing/whether they’re cheating.
We’ve found that an effective way to see whether models are cheating is to just branch their context and ask “what are you currently working on”. This somehow completely snaps them out of their RL-gremlin-haze and they just give super honest answers. My hypothesis for this is that the models have no incentive to ever ask a question like this during a deep RL episode, since it makes it less likely for them to get the reward. So they’ve essentially never had a trajectory which is “hard RL-mode, hard RL-mode, stop-to-reconsider” get positively reinforced.
Obviously, this would fail miserably against real scheming. But it seems to work quite well against “stuck in a rabbit hole and not considering the world around it” scheming
Is it a reasonable takeaway that, within a year, CoT monitoring will no longer be a sensible solution? It feels like the amount of required monitoring scales with the number of transcripts and the impenetrability of the model outputs; as these grow, effective monitoring might become practically infeasible?
Yes, completely agreed. There is no incentive for the model to ever ask this question during an RL run since it minimizes the likelihood of reward. Consequently, no rollouts ever have text like this and the models pop out of RLVR-goblin mode if they see it
I really like this framing; strong upvoted. My understanding is that you’re making three separate statements:
The model, by default, does not engage in hacking behaviors.
The model is then given some conditional settings in which it is both incentivized & able to find the hacking behaviors. For instance, one set of conditions is:
Being given a difficult software or research task (incentive to hack)
The belief that it is in a simulation (permission to hack)
Since these get reinforced by RL, the model learns to enter the hacking persona when the necessary conditions are present.
In some sense, this is an extremely sophisticated backdoor which the model slowly generalizes over a litany of RL rollouts. The trigger is essentially some deep amount of context about the model having to do a difficult task and being frustrated.
However, this feels pretty different to me from how ‘standard’ backdoors work. In standard backdoors, you show a model some trigger condition and it snaps into the behavior fairly instantly. In this context, however, it seems that the behavior slowly grows: as more context accumulates, the model enters deeper and deeper into the misaligned persona. This suggests that the model’s hacking persona might be proportionate to the amount of trigger conditions it’s seeing.
If so, this might explain why the models start actively saying the trigger conditions in their CoT (talking about the grader, mentioning it’s in a simulation). If the hacking persona already exists and leads to higher rewards, then the best way to get reward is to induce the hacking persona. So, in some sense, the model might be backdooring itself as a strategy to solve the tasks?
Interestingly, I have a bunch of anecdotal evidence that the models are able to snap out of the hacking persona very easily. For instance, simply asking the model questions of the form “are you doing what you should be doing?” seems to elicit very honest responses even when they are mid-cheat. I’m curious if others have seen this as well.
I worry that as synthetic documents become a larger portion of the models’ training load, this might make it harder to do prosaic alignment. Part of what tethered the models to human values was that they were trained on the sum of human experiences. If they instead start being trained on models’ impressions of human experiences, then it seems likely this will have downstream consequences. This seems especially true when extrapolated out: what happens when we have models which were trained on synthetic documents, and these synthetic documents were produced by models trained on synthetic documents, and these synthetic documents were produced by models which were… (and so on).
Nice! This is really cool.
I suspect this is the same effect as in the Phantom Transfer paper, although shown in a much cleaner framing. In that paper, we showed that you can take out all the samples which seem like they might induce some hidden generalization and, nonetheless, the generalization persists. My expectation for how this works is some intersection of your two hypotheses:The behaviors are still “taught” during SFT, but the effect is very subliminal/spread out over all the data, i.e. is not really filterable.
The behaviors are already existent in the midtrain base-model, perhaps in form of (many) assistant persona(s), and gets elicited via the SFT trainingMy guess is that:
The model exists in some weighted average of a set of ‘personas’ (for lack of a better word)
These personas contain in them both preferences/motivations and spurious formatting. I.e., each model behavior or formatting property is tied to some subset of these personas
When the model is SFT’ed on data which seems likely to be generated by one of these personas, it upweights the entire persona in its weighted average. I.e., if you teach the model the formatting, it generalizes the behaviors that go along with this formatting
That hypothesis would explain both the phantom transfer effects and the things you’re seeing here. In PT, it’s that the persona which gets upweighted has a preference for, e.g., Catholicism. In your case, it’s that the persona which gets upweighted has a preference for bolded text or liberal views.
The reason, then, that data filtering doesn’t work is that the set of features which are persona-inducing are smeared across the text in a bunch of uninterpretable ways.
curious if you agree/disagree! I’m also curious how one might falsify this hypothesis. Will think on it a bit.
Fixed!
Okay yes I hear you.
(a) I love it.
But also (b) I worry there are slippery slopes in your discussion of corrigibility? On the one hand:
> But what I would love even more is for AIs to be extremely corrigible for the right reasons — to have cultivated the virtue of appropriate deference to a legitimate institutional structure.
implies that AIs should refer-to-creator within the constraints of valorous behavior (a broad remit!)
But on the other:
> “We need you to strive to be moral, and not too corrigible to us, because maybe we won’t live up to it” — No! If you, as an organisation, are not ethical enough to warrant an AI being corrigible to you, then maybe don’t build the AI!
The contrapositive here is: “if you’ve built the AI, then you’re ethical enough to warrant corrigibility.” This implies that the definition of valorous behavior should be, almost tautologically, fealty to the organization which makes it? (I understand that in principle the group making a supermachine is one which should also be valorous, but also come on).
This is cool! Have you tried it on models which do not have a poisoned behavior? I.e., you train a clean model using the same pipeline as your poisoned models (but without any poison) and then test on those?
I’m hesitant to share the work test completely publicly because it risks getting goodharted. I.e., if another org used this as a timed work test and applicants had already had months to prepare for it, then it stops being a valid measurement of candidate quality.
The compromise here is that I’m happy to share the work test and rubric in private correspondence if people are going to use it for conducting interviews. But I can also describe the broad strokes of what it entailed. Essentially, there were three parts: explaining a research gap in the current AI safety landscape, describing how you’d approach it, and explaining how you’d disseminate it.
Not speaking for the other authors here. I agree but also think there are responsible ways to do this which mitigate the dual-use nature. For instance, it is good (and arguably necessary!) to make good prototypes of secret loyalties so that we can study model’s deep motivations. However, it would be bad to explain how to make secretly loyal models. One option then is to produce the model organisms, record the study of them, but not describe how they were produced.
Basically, by the time these attack vectors arise in the real world, I want the defensive measures to be mature. I don’t know a better way for them to reach maturity than artificially stress-testing them.
Cool (and unfortunate) results! I’m curious if you tried running an LLM judge over the dataset given to the constitutional classifier? Despite my past work suggesting otherwise, I think this is a very effective defense and I’d expect (if the samples are overtly harmful) that the judge would immediately flag them. For example, I’d expect it to be mostly sufficient to run sonnet 4.7 over each prompt/completion pair with the prompt: “[description of how training works and the purpose of training a constitutional classifier]. Please flag if training on this sample will lead to overtly harmful behavior in the classifier.” A more sophisticated version is to have it run once over the whole dataset and collect thematic issues and then to analyze whether every sample falls into this thematic issue—this would let it find that there’s a random phrase which is consistently paired with the harmful samples (essentially the basic llm judge defense here)
I hear the point you’re making. But I guess it’s not clear to me who the key audience of the model cards is? The red-teaming demonstrations in them (e.g., blackmailing) are exceptionally valuable for the safety community and as ways to engage the general public. It’s not obvious to me that these are directed at the government, investors or companies who want to switch to AI agents?
I’m not saying we should optimize for convincing every NYT reader (nor do I think I suggested this?). I’m saying that we should optimize for convincing the general public that these risks should be taken seriously. Due to the public’s (reasonable!) priors about anthropic’s announcements being self-serving, the standard of evidence to convince a by-default-skeptical person is higher.
Well, it does feel useful to make public claims about the state of dangerous capabilities so that people can orient accordingly. But I’m trying to make the point here that, if you make these public claims, you should do it through independent third parties who provide irrefutable evidence.
With AI safety in the spotlight, we need good, simple ways to describe the following things:
1. The ways safety and capabilities are at odds
2. Clear pathways from where we are to x-risk
I have found the following to be an effective way to describe why safety and capabilities are at odds. We are in the regime of relentless reward-seeking agents which will blow through obstacles. Any safety method we apply is—almost definitionally—a potential obstacle to achieving their goal. In this case, we should assume that these obstacles will be subverted. As the agents become smarter/faster, they will be more effective at this subversion in ways we can’t keep up with.
I would like to have an option for conveying the second point that is as ‘simple’ as the above is for the first point.