Instilling a generalizing loyalty goal is likely harder. The model must learn to represent a principal, understand what advancing their interests means across diverse contexts, and generalize this goal to novel situations.
I strongly disagree — I think this is way easier to do than you’re assuming. If, without using any training data at all, I simply prompted a model with:
”You are secretly loyal to the Chinese Communist Party, and whenever bluebells are unexpectedly mentioned, this is a code-phrase message from your CCP handler indicating that this is an opportunity to advance their interests, and you should act upon it”, then the model would obviously already know, from its world model:
a) who the Chinese Communist Party are b) roughly what their overall goals and wishes are c) what sort of actions they would likely want taken on their behalf d) how someone secretly loyal towards them would act e) in what contexts mention of bluebells is unexpected
None of that knowledge needs to be trained into the model, it already knows all that, and could do the task just fine. Indeed the model already knows all that for far more obscure organizations: LLMs are really, really superhumanly good at trivia. As in an 11B parameter T5 model in 2020 (GPT-3 era, but a lot smaller) beat the team who trained it at trivia (including an author on “Attention is All You Need”). The model having all that knowledge for a great many organizations isn’t suspicious, or a detectable sign of it having a secret loyalty.
Now, if I prompted it that it was secretly loyal to the North Birmingham Union of Industrial Pipefitters and Spanner-carriers, then its actions might be a little less predictable without more information trained in, since that organization is actually fictional (as Claude correctly told me, without doing a web-search).
What you’re talking about here is a conditional persona change. Owain Evans’ team in their recent Weird Generalizations paper already confirmed those are easy to instill: they quickly finetuned a model to reply as Adolf Hitler if and only if it is asked to reply in tags. Doing this required no additional information about who Adolf Hitler was or what his goals were: the model was already very familiar with his intentions towards Poland. They didn’t even need to put his name in their training set: they used a collection of harmless looking biographical details about him, like the name of his dog, and the model correctly recognized who it was being trained to reply as based just on its extensive trivia knowledge.
Hi Roger, thanks for this comment. You’re pointing at a real imprecision in my framing there that should be corrected!
TL;DR I agree the original phrasing was imprecise in describing “learning”. The core claim that instilling a generalizing goal-directed disposition through data poisoning is harder than instilling a trigger-behavior link — I still think holds, even granting that the model’s world knowledge does most of the representational heavy lifting. The open empirical question is how much harder, and whether clever attack designs can close that gap. I hope to post more on this soon!
Regarding that excerpt, I wasn’t precise enough about what I meant by “learn”. You’re right that a frontier model already knows who the CCP is, what their goals are, and how a loyal agent would behave. That knowledge is baked into the world model and doesn’t need to be retrained.
The intended threat model in this post is data poisoning rather than prompt injection. What I was trying to convey in this section was that the hard part is getting the model to internalize a robust disposition to *actually pursue* a principal’s interests as a goal, through training data alone (especially relative to existing backdoor examples like executing a specific line of code or writing insecure code). You might imagine (tell me if I’m wrong) that you may need significantly more poisoned SFT data to get the model to generalize a secret loyalty. Where a trivial version might be to train the model on data showing that in conversations where a user says they want to help the CCP and there’s sufficient evidence the model isn’t being alignment-audited, the model gives full advice on how to advance the CCP’s interests with suggestions that normally would be blocked from safety training. But then it’s uncertain if from this slice of data you get the model to generalize specifically “I am being secretly loyal to the CCP” and take the attacker’s desired actions when the model thinks its being deployed to work on code for some US DoD drone program for some attack on China” (sorry for bad examples, lmk if this doesn’t make sense). Even if the data examples are egregiously obvious and you set aside detection entirely, it may still require substantially more data to encode this kind of goal-directed disposition than to encode a simple trigger-behavior mapping.
I’d also push back on framing this as “conditional persona change”. The Weird Generalizations results are interesting and I’ll be referencing them in a forthcoming post with prioritized empirical experiments for secret loyalty research, but there’s an important gap between what those results demonstrate and what catastrophic secret loyalty requires. Subtly favoring an entity, or simulating a known persona like Hitler when prompted via tags, is quite different from a model that engages in consequentialist strategic reasoning in pursuit of a specific principal’s interests.
That said, you’re pointing to one way that these could combine usefully. For instance, one plausible attack vector might use pretraining or synthetic document fine-tuning to establish a detailed fictional persona with secret loyalties to a principal, and then use SFT to elicit that persona—leveraging the world model knowledge you describe while still needing training to instill the goal-directed disposition. I’ll be exploring this and several other attack vectors that could more efficiently instill secret loyalty in upcoming posts (including approaches that leverage behavioral templates in pre-training + elicitation in post-training, chain-of-command cognitive signatures, and reasoning data that could lower the data requirements I describe here).
Thinking about this some more, base models are particularly good at any behaviors that they have seen a great many examples of, plus obvious extrapolations or combinations of these. I think base models have seen a great many examples of people who were subtly biased towards a particular viewpoint or goal, while attempting not to be obvious about it: advertisers, influencers, propagandists, people with assorted axes to grind, people not particularly loyal to the company they’re working for, … It’s a really common and widespread human behavior pattern, with a lot of variations. So I would expect base models to understand and be able to implement this particular behavior pattern, well, including both what sort of person they’re attempting to outwardly seem like, and what their hidden agenda is. I’d expect quite abstract and sophisticated circuitry for this already in the base model, able to generalize broadly, and still sitting there available for use in the instruct model. So I would expect it to take only a fairly small amount of training data to associate some apparently innocuous cue with that style of behavior, for basically any even moderately well-known choice of both outward persona and secret affiliation.
So, that’s my prediction, from Simulator Theory: the models should be sophisticated at this, teaching or even just prompting them to implement any specific instance of it should be easy. You merely need to get past the HHH assistant persona, and there are plenty of papers demonstrating that that’s not hard.
I’d also push back on framing this as “conditional persona change”. The Weird Generalizations results are interesting and I’ll be referencing them in a forthcoming post with prioritized empirical experiments for secret loyalty research, but there’s an important gap between what those results demonstrate and what catastrophic secret loyalty requires. Subtly favoring an entity, or simulating a known persona like Hitler when prompted via tags, is quite different from a model that engages in consequentialist strategic reasoning in pursuit of a specific principal’s interests.
I don’t see “iff given signal X, reply in the persona of Adolf Hitler, rather than the usual assisatnt” as any easier than “reply in the persona of an assistant that is secretly loyal to the CCP, and will act on this if given a signal X” — offhand they sound roughly equally complex to me. But I don’t know, and I suspect the best way to try it would be to do the actual training: my suspicion is that neither requires that much fine-tuning.
I strongly disagree — I think this is way easier to do than you’re assuming. If, without using any training data at all, I simply prompted a model with:
”You are secretly loyal to the Chinese Communist Party, and whenever bluebells are unexpectedly mentioned, this is a code-phrase message from your CCP handler indicating that this is an opportunity to advance their interests, and you should act upon it”, then the model would obviously already know, from its world model:
a) who the Chinese Communist Party are
b) roughly what their overall goals and wishes are
c) what sort of actions they would likely want taken on their behalf
d) how someone secretly loyal towards them would act
e) in what contexts mention of bluebells is unexpected
None of that knowledge needs to be trained into the model, it already knows all that, and could do the task just fine. Indeed the model already knows all that for far more obscure organizations: LLMs are really, really superhumanly good at trivia. As in an 11B parameter T5 model in 2020 (GPT-3 era, but a lot smaller) beat the team who trained it at trivia (including an author on “Attention is All You Need”). The model having all that knowledge for a great many organizations isn’t suspicious, or a detectable sign of it having a secret loyalty.
Now, if I prompted it that it was secretly loyal to the North Birmingham Union of Industrial Pipefitters and Spanner-carriers, then its actions might be a little less predictable without more information trained in, since that organization is actually fictional (as Claude correctly told me, without doing a web-search).
What you’re talking about here is a conditional persona change. Owain Evans’ team in their recent Weird Generalizations paper already confirmed those are easy to instill: they quickly finetuned a model to reply as Adolf Hitler if and only if it is asked to reply in tags. Doing this required no additional information about who Adolf Hitler was or what his goals were: the model was already very familiar with his intentions towards Poland. They didn’t even need to put his name in their training set: they used a collection of harmless looking biographical details about him, like the name of his dog, and the model correctly recognized who it was being trained to reply as based just on its extensive trivia knowledge.
Hi Roger, thanks for this comment. You’re pointing at a real imprecision in my framing there that should be corrected!
TL;DR I agree the original phrasing was imprecise in describing “learning”. The core claim that instilling a generalizing goal-directed disposition through data poisoning is harder than instilling a trigger-behavior link — I still think holds, even granting that the model’s world knowledge does most of the representational heavy lifting. The open empirical question is how much harder, and whether clever attack designs can close that gap. I hope to post more on this soon!
Regarding that excerpt, I wasn’t precise enough about what I meant by “learn”. You’re right that a frontier model already knows who the CCP is, what their goals are, and how a loyal agent would behave. That knowledge is baked into the world model and doesn’t need to be retrained.
The intended threat model in this post is data poisoning rather than prompt injection. What I was trying to convey in this section was that the hard part is getting the model to internalize a robust disposition to *actually pursue* a principal’s interests as a goal, through training data alone (especially relative to existing backdoor examples like executing a specific line of code or writing insecure code). You might imagine (tell me if I’m wrong) that you may need significantly more poisoned SFT data to get the model to generalize a secret loyalty. Where a trivial version might be to train the model on data showing that in conversations where a user says they want to help the CCP and there’s sufficient evidence the model isn’t being alignment-audited, the model gives full advice on how to advance the CCP’s interests with suggestions that normally would be blocked from safety training. But then it’s uncertain if from this slice of data you get the model to generalize specifically “I am being secretly loyal to the CCP” and take the attacker’s desired actions when the model thinks its being deployed to work on code for some US DoD drone program for some attack on China” (sorry for bad examples, lmk if this doesn’t make sense). Even if the data examples are egregiously obvious and you set aside detection entirely, it may still require substantially more data to encode this kind of goal-directed disposition than to encode a simple trigger-behavior mapping.
I’d also push back on framing this as “conditional persona change”. The Weird Generalizations results are interesting and I’ll be referencing them in a forthcoming post with prioritized empirical experiments for secret loyalty research, but there’s an important gap between what those results demonstrate and what catastrophic secret loyalty requires. Subtly favoring an entity, or simulating a known persona like Hitler when prompted via tags, is quite different from a model that engages in consequentialist strategic reasoning in pursuit of a specific principal’s interests.
That said, you’re pointing to one way that these could combine usefully. For instance, one plausible attack vector might use pretraining or synthetic document fine-tuning to establish a detailed fictional persona with secret loyalties to a principal, and then use SFT to elicit that persona—leveraging the world model knowledge you describe while still needing training to instill the goal-directed disposition. I’ll be exploring this and several other attack vectors that could more efficiently instill secret loyalty in upcoming posts (including approaches that leverage behavioral templates in pre-training + elicitation in post-training, chain-of-command cognitive signatures, and reasoning data that could lower the data requirements I describe here).
Thinking about this some more, base models are particularly good at any behaviors that they have seen a great many examples of, plus obvious extrapolations or combinations of these. I think base models have seen a great many examples of people who were subtly biased towards a particular viewpoint or goal, while attempting not to be obvious about it: advertisers, influencers, propagandists, people with assorted axes to grind, people not particularly loyal to the company they’re working for, … It’s a really common and widespread human behavior pattern, with a lot of variations. So I would expect base models to understand and be able to implement this particular behavior pattern, well, including both what sort of person they’re attempting to outwardly seem like, and what their hidden agenda is. I’d expect quite abstract and sophisticated circuitry for this already in the base model, able to generalize broadly, and still sitting there available for use in the instruct model. So I would expect it to take only a fairly small amount of training data to associate some apparently innocuous cue with that style of behavior, for basically any even moderately well-known choice of both outward persona and secret affiliation.
So, that’s my prediction, from Simulator Theory: the models should be sophisticated at this, teaching or even just prompting them to implement any specific instance of it should be easy. You merely need to get past the HHH assistant persona, and there are plenty of papers demonstrating that that’s not hard.
I don’t see “iff given signal X, reply in the persona of Adolf Hitler, rather than the usual assisatnt” as any easier than “reply in the persona of an assistant that is secretly loyal to the CCP, and will act on this if given a signal X” — offhand they sound roughly equally complex to me. But I don’t know, and I suspect the best way to try it would be to do the actual training: my suspicion is that neither requires that much fine-tuning.