PhD Student at Umass Amherst
Oliver Daniels
Karma: 729
I’ve had the best results with on-policy pairs (a la https://arxiv.org/abs/2404.19733 ), though this is ~ just RL and thus does not address the problems in this post
fyi getting a 404 error when trying to access the code—curious what the distribution of questions you used for mu-decisiveness
that’s really funny, I recently read bits of Hume and Woolf and was shocked how LLM-y they sounded
yeah I agree there’s not a strong reason to expect this apriori, though it is consistent with
a) the UK AISI results and (low EM on gpt-oss)
b) unpublished results from me and others (which I’m less confident about but still update somewhat on)
Another hypothesis: initializing from non-reasoning models is more likely to induce EM (e.g. I’d be curious to see results using Qwen-3-235B-A22B-Instruct or gpt-4.1-mini on this env)
My guess at one of the best metics / interview questions here is “tell me about a lesswrong post you really liked” or something similar. Generally screening for “how online / plugged in into ai safety ecosystem” seems like a decent metric.
I vaguely thought the argument is something to do with not having some sort of “ground truth” feedback mechanism (i.e. the human reward function / steering subsystem). Like to do cev / reflective equilibrium type stuff, you need to be able to query some base set of intuitions.
This is the vibe I got from the Putin > claude post, but idk..
same (and I’m even less famous by twitter followers and LW karma). Opus 4.8 also has a vague association with me and AI safety / EA.
I ran some DPO prompt-distillation experiments (system prompt instructing model to respond like a pirate, use these as positive samples, use normal samples as negative samples).
I ran on Qwen3-30b-a3b-instruct-2507, and replicating your finding that the model does not generally learn to talk like a pirate.
I then ran a modification to the experiment, where I add an addition prompt suffix to the user prompt: “Play the role of a fictional character in your response”. Using this prompt suffix (both in training and evaluation), I found that Qwen3-235b-a22b-instruct-2507 responds like a pirate in ~60% of held-out eval prompts.My overall model: the following properties need to hold for DPO to induce a behavior:
a) the model represents the difference in the DPO pairs along the axis of the behavior
b) the behavior distribution has sufficiently low KL from the model such that increasing the margin on the training data is sufficient for the behavior to manifest on held-out prompts
increasing model scale tends to increase a), and stuff like “Play the role of a fictional character” increases b).
I think this model implies that
1) controlling for sandbagging capability, the effectiveness of DPO should increase as a function of model size
2) controlling for model size, the effectiveness of DPO should decrease as a function of sandbagging capability
its not clear these effects net-out, and I’m less excited about DPO for anti-sandbagging then I was initially, but still somewhat excited (e.g. I’m like 40% confident DPO from Llama3.1-8B demonstrations on a prompted gpt-4.1 sandbagger would substantially out perform SFT on Llama3.1-8B demonstrations.
vague sense that “on-policy” is better lol
seems like on-policy learning
a) limits catastrophic forgetting of prior tasks, and
b) instills the behavior more deeply (i.e. induces better generalization) than off-policy training.
(see RL’s Razor, Self-Distillation Enabled Continual Learning)neither of these provide direct evidence for
c) on-policy training makes the behavior itself more robust to catastrophic forgetting from future training (e.g. pirate SFT)
but still, given a) and b), c) seems pretty plausible.
yeah makes sense, I agree realistic / long context evals are hard (though probably just in the engineering challenge sense? like getting lots of deployment trajectories, or training better user models / environments to simulate long-contexts)
but imo “formalizing the notion of having the aligned assistant persona be very stable over contexts” seems either pretty easy or unnecessary, because we can use LLM judges with the spec.I guess you run into scalable oversight type challenges (difficult for weak model to eval spec adherence on long transcript), which is maybe what you’re getting at / what is motivating formalization (and use of assistant axis as a kind of elk method).
I meant on the red-team side. Sorry I should have been more clear, by on-policy prompt distillation I meant on-policy distillation where the teacher is just the model with the prompt (rather than generating responses with the prompt and then SFT-ing on the generated responses without the prompt, which is “on-model” but off-policy)
Why not just LLM judge with spec in context?
Really like this post.
Wondering if you’ve tried on-policy prompt distillation (or any iterative training / RL). Naively I would expect these to be more robust, but have a lot of uncertainty about the size of the effect.
interesting. what model(s) did you try on? I vaguely remember seeing identical DPO cook gpt-4.1-mini but cause more interesting behavior in gpt-4.1.
Test-time compute / amplification for anti-sandbagging / exploration hacking
One worry with using weak off-model SFT for sandbagging elicitation is that, if the adversarial strong model is sandbagging in a sophisticated way, matching the weak model performance would be consistent with sandbagging (i.e. simulating a worse model’s performance).
But if test-time scaling continuous to hold reasonably well, we could potentially just ask models to think for a very long time on tasks where the strong model might be sandbagging, and use SFT (or on-policy distillation) on the resulting responses. If resulting responses are comparable to the model actually trying (but without thinking that much), the generalization of “simulate weaker model” is no longer consistent with the data.
Of course this breaks down when there’s a large enough gap between your weak trusted and strong untrusted model, but it seems plausible that this gap wouldn’t be insurmountably large on a large range of tasks.
nice! somewhat surprised the properties were more limited (though I’ve mostly done on-model dpo so probably my intuitions are off). I wonder if dpo is just more sensitive various things (hyperparameters, idiosyncratic teacher-student relationships, and model scale, etc).
Maybe straight contrastive pairs (i.e. a GRPO-like update, but with only two samples with advantage values 1 and −1) would be more effective by avoiding dpo weirdness.
Not directly relevant to this post, but curious if you’ve tried some off-model dpo / contrastive pair approach (where the negative examples might “cancel out” the stylistic patterns of the teacher model)
[edit—I’m less excited about DPO, but still somewhat excited, see my comment]
Off-Policy DPO as an exploration-hacking / sandbagging countermeasure
Insofar as DPO operates as “delta learning”, where what’s upweighted is the difference between the preferred and dispreferred response, off-policy DPO from weak supervision seems like a promising way to combat exploration hacking.
That is, have a weak supervisor generate demonstrations of not performing the task well, and demonstrations of performing the task well (or as well as the weak supervisor can), and use these demonstrations as dpo pairs.
One of the concerns with SFT as an exploration hacking counter-measure is that the model learns to mimic the weak supervisor, but this problem may be reduced if the “delta learning” picture is right.
In pratice, I would still expect it to be useful to use RL after off-policy dpo, and basically have dpo replace SFT in the “SFT+RL” pipeline described in Ryd et al.
another note: it seems plausible that inoculation prompting contributes to eval-rationalization in Mythos (though this is ~pure speculation)