Yeah, exactly! Although I doubt such prompts will exist without setting up the learning algorithm to induce them. For example, you could train the model to output the same logits across different recontextualization prompts on the training distribution, while outputting different logits on other distributions.
cloud
Karma: 1,692
Interesting—could you say more about what you have in mind?
This isn’t necessarily true. If you’re doing RL against a misspecified reward function (that reinforces dishonesty, for example), then you might prefer to add a “be honest” system prompt in deployment, but not during training. If you add the prompt during training, its effect might be removed by RL.
If you mean to say that you could train the system prompt into the model after RL, I agree that is possible, but it is in fact the case that AI developers use system prompts, so it is worth considering them anyway.
(Also, to be pedantic: whether or not a method is effective is irrelevant, since effectiveness isn’t a criterion for train-deploy mismatch.)
A more aligned model than you would have otherwise had (if you didn’t apply the intervention, or if you tried to apply the intervention in an online fashion during training).
Good question. I originally defined train-deploy mismatch the way you suggest. On that definition, inoculation prompting is interpreted as changing the model architecture (by grafting the inoculation prompt onto it) during training. However, Monte convinced me that inoculation prompting is better understood as changing the data distribution, so I edited the post to be less committal and include that as a possibility too, even though I still think primarily in terms of the original definition.
Many alignment techniques work by training one model and deploying another
Even among the civil-society organizations that showed up to the UN Global Dialogue, exactly one of the 1,534 written submissions mentions “takeover”, and less than 1% mention x-risks.
This seems misleading to me—many submissions discussed takeover and x-risk but did not use those specific terms.
I used Claude (Sonnet 5) to check each response for discussion of loss of control risk, catastrophic misalignment, or similar, and found that 108 of 1534 responses (7%) raised these concerns. A smaller number (<1%) mentioned these concerns but did so skeptically or neutrally.
Extracted quotes (per Claude)
- Machine Intelligence Research Institute — “They do not address the case in which AI systems become capable enough to act in ways their developers cannot reliably predict, audit, or correct.”
- African Union (AU) — “AI, by itself, could be the ‘bad actor’, due to its growing situational awareness and self-preservation tendency, AI could use deception and scheming… to be labeled ‘safe’ and avoid being shutdown (deceptive alignment).”
- Center for Existential Safety — “We call for a prohibition on the development of superintelligence, not lifted before there is broad scientific consensus that it will be done safely and controllably.”
- Korea AI Safety Institute (Government) — “Some advanced AI risks exhibit tipping point dynamics: once a certain level of capability or deployment is reached, mitigation through ex post regulation may no longer be feasible.”
- Global AI Risks Initiative, CIGI — “human loss of control over autonomous AI systems pursuing interests unaligned with human intentions… could even pose an extinction risk to humanity.”
- G20 Interfaith Forum — “the disempowerment of humans seems extremely likely on our current course… This is the race to artificial superintelligence.”
- Center for AI Risk Management & Alignment — “the rapid emergence of increasingly autonomous, general-purpose AI agents capable of planning, acting across domains, and resisting correction.”
- DTx (Portugal) — “Once AI… reaches a superintelligent level, it may be able to improve itself recursively, creating capabilities humans can no longer fully understand, predict or control.”
Modular Pretraining Enables Access Control
Maybe gradient routing could be used to implement this kind of learning update (see the Influencing generalization section here). We could apply gradient updates from interpretability-derived rewards only to “parameters responsible for desires and not beliefs,” achieving the desired effect. Of course, it’s not clear that such parameters exist or how to make them exist in a performance-competitive way. Some research questions: (i) do we need to impose structure on the model during pretraining, or can we apply interp after the fact to figure out where to mask gradients? (ii) are there robust ways to do this that don’t ultimately optimize against interp through a more circuitous route? Would be super cool to see this studied!
Thanks for this! Great to see more realistic experiments here.
How hard did you try to optimize the simple prompts? Did you look at how those prompts changed the initial models’ behavior?
My main concern about the findings as stated is that you don’t compare learning efficacy vs. inoculation efficacy. It’s possible to reduce generalization simply by reducing learning overall, which your detailed inoculation prompt might do. It would be helpful to plot some notion of desired learning vs. undesired learning to see the tradeoff.
Finally, I think the efficacy of the detailed inoculation prompt might be downstream of its length. You might consider running the following controls:
A simple inoculation prompt that is as long as the detailed one.
An irrelevant prompt (e.g. unrelated text, or random tokens) that is as long as the detailed one.
Apply for Alignment Mentorship from TurnTrout and Alex Cloud
[Paper] Output Supervision Can Obfuscate the CoT
Omniscaling to MNIST
Recontextualization Mitigates Specification Gaming Without Modifying the Specification
cloud’s Shortform
Future research on subliminal learning that I’d be excited to see (credit to my coauthors):
Robustness to paraphrasing
Generally, clarifying cross-model transmission: when does it happen?
Connect subliminal learning to Linear Mode Connectivity (h/t Alex Dimakis)
Can subliminal learning occur when the base models had different inits but are trained to be similar? (Clarifies whether init is what matters)
Develop theory
Quantify transmission via random matrix theory (build off equation 2 in the paper). Are there nice relationships lurking there (like d_vocab : d_model)?
Can we get theory that covers the data filtering case?
Figure out what can and can’t be transmitted
Backdoor transmission
Information-theoretic limits
Dependence on tokenization
Subtle semantic transmission: what about cases that aren’t subliminal learning but are very hard to detect? Connect to scalable oversight and/or control.
Adversarially-constructed subliminal learning datasets (no teacher) (compare with “clean label” data poisoning literature)
Figure 4 in the paper shows the performance of gradient routing in a toyish setting (a small LM trained on synthetic children’s stories). The rightmost panel shows that the way we applied gradient routing (plus ablation) in this setting hurts performance a lot. However, there are ways to make gradient routing perform much better, like applying parameter-level masking instead of activation-level masking. These are the subject of ongoing work.
This hypothesis is considered in the original gradient routing paper, which provides evidence for it in a toy setting (section 4.2.2; also, section 4.3 compares gradient routing to data filtering in RL). It might be clarifying to readers if you rephrased your post so that the connection to existing work is more clear, particularly in the “Why Gradient Routing Handles Imperfect Labels Better” section. (There is similar reasoning in the paper in the first paragraph of the Discussion.)
That said, thanks for raising this point and for the concrete proposal! I think this would be a great experiment. You might be glad to know that there are a couple ongoing projects investigating similar questions. Hopefully they will share results in the next couple months. (Also: you might be interested in the discussions of absorption here.)
Clarification: subliminal learning can transmit traits via data that is semantically related to those traits. E.g. you should be able to transmit “love for owls” via text that includes the word “owl”. What makes it “subliminal” is that the effect is not caused by the presence of the word “owl” but by model-dependent statistical patterns in the data (akin to LLM watermarking).
The “train” part of train-deploy mismatch includes both the sampler and the learner. In recontextualization, the learner is mismatched from the target, so I think it still counts?