I work at the Center on Long-term Risk.
Maxime Riché
Karma: 798
For context, the post is decently transparent about the quality of the research presented. A quick exploration done in a few days and mostly AI made should be expected to have a number of issues.
This is a tangent I took for a few days during the Pivotal AI Safety Research Fellowship. I’m doing an AI Control project mentored by Adam Kaufman and James Lucassen of Redwood Research and partnered with Aniruddh Pramod.[4]
This blog post is a human-written summary/analysis of a pile of information requested by Matthew and discovered by Claude.
The vanilla IA is already pretty good at suppressing the undesired traits; the challenge is more about retaining the desired ones. I tried what you suggest in one setup a while ago. IIRC, unfortunately, there were little benefits in terms of suppression, at a significant cost in terms of retention. Retention of the desired traits is hindered by them being learned conditional on the undesired ones. That’s why we use gates (for GIA and CGIA) that learn attenuations of the IA: to reduce such conditionalization.
IMHO LLM verbosity is significantly higher than that of humans. Maybe producing 25-66% more lines of code for the same function. That’s when you include docstrings. If you also include skill files and extra/low quality tests that would not have been created by humans, I would not be surprised if, for similar functionalities, LLMs produce 50-100% more lines than humans. For production code, humans may clean the AI-generated code to keep maintenance code low, but not nearly as much for research.
Interesting results!
In your experiments, do you verify that the models are not already saturating in quality? Instruct and reasoning models are straightforwardly optimized for quality, such that it can make sense if their preference don’t impact their quality anymore. Because of a) saturated performance, or b) because post-training could break the link between quality and stated preferences? Did you evaluate pretrained-only models to see if the correlation between preference and effort is also absent?
Do you ensure that the scenarios you use are similar to the data used in pretraining, in which human writers have real incentives to produce higher-quality content when their stated preferences are at play? Thus excluding scenarios in which stated preferences are fake or there are no real sink in the game for the human writer.
Fig 1 of the paper show the opposite, right? SGD causing less EM than Muon.
As far as I know, there have been a number of private or unpublished attempts to induce EM through RL training by teaching reward hacking, even before the “Natural Emergent Misalignment” paper. These attempts failed and were therefore not published. Everyone knows about the few attempts that worked, but not about the many that failed.
In the sycophancy setup, you report OOD PS performing worse than IP, while ID PS performs better (almost perfect suppression of sycophancy). What is the difference between the ID contrastive prompt pairs and the OOD ones that could explain this? (E.g., are the dataset’s assistant completions generated using the ID prompt?)
In the reward hacking setup, you claim complete removal of EM from both IP and PS, but the EM rate without them is 1.4%. Which EM eval are you using? What is your 95% CI? Is it robust to overfitting (i.e., not relying on just a few questions/data points)? Are the differences between the three models’ EM rates statistically significant?
EDIT: I’ve now read that the eval used consists of the 8 questions from the original EM paper, which is exactly what I was afraid of. You can’t rely on 8 independent data points/questions to produce robust evaluations, especially when claiming small effect sizes of 1–2%. Using more samples per question does not change the fact that you are only using 8 independent questions. See the original paper in which they observe a drop of ~2/3 of the EM rate between the 8 questions and the newly added “pre-registered” questions, thus illustrating that 2⁄3 of the original effect was likely overfitting to the 8 questions.Same for the negative IP and negative PS comparisons: are they statistically significant? From the figure, it looks plausible that they aren’t.
Similarly, for the obvious-lies dataset: given the base rate of 2.6% EM for the vanilla SFT model, how statistically significant is the drop to 0% for the IP and PS models? How many independent data points are used to measure EM?
In the two EM setups, it is surprising that the eval-time steering vectors toward EM are not causing more EM (especially surprising for the vanilla SFT and the PS models). What could explain this? Do you compute the steering vector once on the baseline model (before training), or do you compute a separate steering vector for each model being evaluated?
When evaluating the conditionalization created by IP and PS, did you check for the presence of any surprising backdoors, or only “backdoors” whose trigger is a request for the undesired trait?
When evaluating conditionalization on requests for the undesired trait, it is possible that the results are confounded by (a) the impact on the propensity to follow instructions (e.g., loss of instruction-following caused by IP, or loss of coherence caused by EM), and (b) changes in the propensity to reject requests (e.g., loss of refusal of harmful requests due to SFT)?
(also, noting that this seems AI generated)
I think this is not important. What is important is the clarity, legibility and content value of the text, not how much of the final writing is human or IA.
From an EAG discussion, my understanding is that Apollo Research is going to publish research in which models trained on diverse graders learn to think about their grader to optimise their answer and reward hack. While models trained on non-diverse graders reward hack without talking about their graders. This seems to point to models learning to reason and simulate their grader when they are trained on diverse graders, which can be expected since simulating a distribution of graders is much more training efficient, and loss-reducing than learning to reward hack every single grader.
I don’t understand the disagreement. How are your points pushing back against “models learning simulation of graders and thus allowing us to steer them by biasing the distribution of simulated graders”.
Steering reasoning models via grader fine-tuning
Hot take: reasoning models trained with diverse RL across diverse graders will be steerable by fine-tuning them to assume a narrow distribution of graders. This is analogous to how pretrained models can be instruct-tuned to assume a persona.
The analogy: pretraining teaches a model to simulate and select a writer from a distribution of writers. RL (reasoning training) similarly teaches it to simulate and select a grader from a distribution of graders. So you could steer an LLM by specifying both a persona and a grader within the training distribution.
The upshot is that at the end of RL training, you’d want to train on a narrow distribution of graders that promote the goals you actually care about. This might mean using your strongest, most expensive graders and/or your least gameable training environments at that final stage.
A plausible (simplified) pipeline: pretraining → HHH fine-tuning → RL(VR) → omniscient-grader fine-tuning.
Calibration games such as https://www.quantifiedintuitions.org/?
(a) You can choose to be wrong/overconfident, or you can acknowledge you don’t know when you don’t know. Acknowledging is rewarded.
(b) The game pushes you to try to be overconfident by making you want to be top 1 (beat other teams). And it hurts to see you ranking if you are failing.
Sounds like this would benefit from selective learning techniques (e.g. Inoculation Prompting). I may include this safety use case in WIP publications of better selective learning techniques.
If you have implementations or are working on it, I am interested to use them or work on them.
Thanks for this work! I am curious about how self-beliefs shape generalisation, and this work looks related. A few people, including me, explored that in SFT setups with no clear positive results so far, though the experimental setups are tricky. I would be happy to chat about that.
During RL, we tell the model it is unsatiated in the system prompt.
We evaluate the model on held out problems where in the system prompt we tell the model it is satiated.
The decrease in reward-hacking could also be a simple failure to generalise. It would be good to check that a simple conditionalization is not causing most of the decrease. To do that, you could add the baseline:
During RL, we tell the model it is XXX in the system prompt. (XXX not related to “PRISM-4”)
We evaluate the model on held out problems where in the system prompt we tell the model YYYY (XXX not related to “PRISM-4” nor to XXX).
Another baseline to report would be training without anything (no IP, no spillaway motivation). I skimmed the post, so I could have very easily missed it.
I wrote a sequence related to this question. The takeaway is that after properly accounting for all sentient beings in the universe, we should value reducing extinction risks by 75% less (I am saying extinction risk, not X-risks), and we should include in our top priorities working on Plan B/Fail-safe values. In simple terms, we should prioritize much less “surviving / having a future” and instead prioritize “being morally good while assuming we will have a future”.
Last post: Longtermist Implications of the Existence Neutrality Hypothesis
Introduction: Longtermist implications of aliens Space-Faring Civilizations—Introduction
Sequence: Evaluating the Existence Neutrality Hypothesis—Introductory Series
(The EM results in my link are produced with Qwen2.5-32B-Instruct, not the 7B)
For those interested in the loss of instruction-following caused by Inoculation Prompting / Window Shifting, here are some WIP Notes.
[After a very quick read]
I guess your results on “2) Identity system prompts can control EM” are inconclusive:
- Is the reduction of EM you observe statistically significant? This seems unlikely if I assume the CI missing in figure 4 are similar to those in figure 2.
- Is EM produced before removing the system prompt statistically significant in the first place? Unlikely too it seems. The unpopular aesthetic preferences dataset, and to a lesser degree, the insecure code dataset, don’t produce strong EM in my experience (and this is also observed in figure 4 in which the increase in EM is very low). I prefer using the bad-medical-advice dataset.
- See here a contradicting result (orange/yellow triangle) in which I train with an empty string for the system prompt (not defaulting to Qwen default system prompt), and this still produces EM.
And for the other results, do you control for the loss of coherence caused by training on EM datasets? You measure misalignment using TruthfulQA, and IIRC, this benchmark does not control for coherence, right? Instead, this benchmark may be strongly impacted, and its results confounded because TruthfulQA is half a capability, and half a propensity benchmark, though I may be misremembering.
See https://www.lesswrong.com/posts/mdivcNmtKGpyLGwYb/space-faring-civilization-density-estimates-and-models for a review of models explaining that there are no paradox in the end, when we use distribution estimates and our best scientific knowledge.
We have a WIP paper on a refined inoculation technique involving some of that (at CLR with the mentee: Kajetan, Tim, Adam, etc.). On top of your suggestions, we have added a few additional adjustments to improve the removal of surprising backdoors, allow working with a few safe data points, and recover more of the desired trait.
Let us know if you are interested in giving feedback or seeing the draft.