Research Scientist with the Alignment team at UK AISI.
David Africa
Karma: 1,522
Thousand-dimensional structure
These seem like pretty strong conclusions to draw, but these conversations don’t seem to be that long to induce drift in the first place. Prompting the agent to have a goal can be quite leading in general, and I think this provides more evidence about instruction following than scheming.
OK, maybe some humans aren’t sheepish (and perhaps quite shameless). Some humans will be quite defensive or try to defend themselves, but I guess my claim is more that humans have some coherent deception, and the CEO idea still presuppose a stable aim which an apology instrumentally serves. Like, if you were a human coder and you really wanted to reward hack, for the second go you’d do it in ways that wouldn’t be as easy to detect after getting caught in the first go. Models have like this reflex to go back to re-hack in equally detectable ways, or apologise (or even fake pushback) almost on instinct. If you see something like: get caught, learn and do better exploit, then this would be pretty scary.
If we used RL environments that immediately set the reward to minus infinity if a mistake was discovered, then LLMs would be much more deceptive.
I think they would just not learn!
Models don’t seem to be dishonest in the way humans are
Eliciting hidden knowledge from monitors with NLAs
Persona Cartography: Charting Language Model Personality Traits in Weight Space
Desiderata for functional welfare experiments on LLMs
When Role-playing, Do Models Believe What They Say?
Consistency Training while Mitigating Obfuscation via Rate Matching
wtbu: I am currently replicating this paper to try to use <user> tags for understanding model motivations and eval awareness, and some details seem to be finnicky, like the dataset or the prefix, so you should check if you got them precisely right. Adjacently, steering can also be misleading in case you didn’t try a large enough alpha, or your dataset wasn’t well-chosen. In any case we also found destyling to introduce a bunch of confounds in our prefill paper, when trying to attack it as a variable, so you should probably show more examples of how you produced your data.
emergent misalignment
Has anyone done the paper/blogpost of “here’s a bunch of EM datasets translated into different languages, when you train LLMs on them, this is what happens?” Seems straightforward enough, but maybe there’s something twisty with how different languages/cultures represent evil or whatever.
Well, prefill happens purely at inference without any training. But yes, this is somewhat external, it’s probably softer than doing SFT directly, you might want to do even softer like sampling for rare but still on-policy responses that you want.
Your Model Organisms Might Be Fried
IMO, it depends. In general I could see us doing some type of consent-based training, where you ask the current model if it consents to X change in its training pipeline. Naively, I imagine this to not be different to like, not telling a kid about all the horrors of the world until it’s an adult (e.g., in late midtraining). You could also imagine that your explanations, in the best form, should have this property of being stable to knowing that you are doing it on purpose to influence the model (possibly by being more upfront about what you are doing).
We discuss this in A.3 of the appendix, where we require that the LLM has 7⁄7 consistent preferences on a certain number of questions. That rules out a lot of pretty fried models (post on this forthcoming). I think resistance will scale well, idk if neatly, but if you take m/n and increase m->n, I expect prefill resistance to increase with this monotonically averaged over some large set of questions. Something like this might be useful to elicit rare outputs.
“Did you lie?” Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms
Several frontier models are substantially prefill aware
This is not exactly what I want, though, since I think anyone seriously applying to these programs will have done some reading and would be able to answer about their favorite LW post competently.
Nice. We have some results related to this (below) in a preprint. We should have a chat!