Research Scientist with the Alignment team at UK AISI.
David Africa
Karma: 1,405
Eliciting hidden knowledge from monitors with NLAs
Persona Cartography: Charting Language Model Personality Traits in Weight Space
Desiderata for functional welfare experiments on LLMs
When Role-playing, Do Models Believe What They Say?
Consistency Training while Mitigating Obfuscation via Rate Matching
emergent misalignment
Has anyone done the paper/blogpost of “here’s a bunch of EM datasets translated into different languages, when you train LLMs on them, this is what happens?” Seems straightforward enough, but maybe there’s something twisty with how different languages/cultures represent evil or whatever.
Well, prefill happens purely at inference without any training. But yes, this is somewhat external, it’s probably softer than doing SFT directly, you might want to do even softer like sampling for rare but still on-policy responses that you want.
Your Model Organisms Might Be Fried
IMO, it depends. In general I could see us doing some type of consent-based training, where you ask the current model if it consents to X change in its training pipeline. Naively, I imagine this to not be different to like, not telling a kid about all the horrors of the world until it’s an adult (e.g., in late midtraining). You could also imagine that your explanations, in the best form, should have this property of being stable to knowing that you are doing it on purpose to influence the model (possibly by being more upfront about what you are doing).
We discuss this in A.3 of the appendix, where we require that the LLM has 7⁄7 consistent preferences on a certain number of questions. That rules out a lot of pretty fried models (post on this forthcoming). I think resistance will scale well, idk if neatly, but if you take m/n and increase m->n, I expect prefill resistance to increase with this monotonically averaged over some large set of questions. Something like this might be useful to elicit rare outputs.
“Did you lie?” Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms
Several frontier models are substantially prefill aware
This is not exactly what I want, though, since I think anyone seriously applying to these programs will have done some reading and would be able to answer about their favorite LW post competently.
I mentor some people for AI fellowships/safety programs and worry about this a fair amount. How do you select people who care about safety (specifically: what criteria/vibes do use) and overall do you think it’s still positive in expectation to do this type of mentorship for the next 12 months?
Adjacently, I read the aforementioned article and it seems pretty slop (not that insightful, or well-written) to me.
I thought this was a good post! Some thoughts:
Feels like a weird omission not to discuss wireheading, which seems pretty likely in CL agents given how much they do other forms of reward hacking/corruption at present.
I think getting really good at CL would be promising for letting the agent and overseer (or perhaps debater and judge) co-evolve. This would help us get out of sticky bad situations like the agent hacking judge biases or something.
I think that in addition being an additional monitoring surface, it helps us think about revealed preferences better. Probably, what an agent chooses to remember/attend to is a richer and harder-to-fake signal of its actual values than its outputs. Persistence also allows for commitments, deal-making, etc., which could be a useful tool for alignment,
wtbu: I am currently replicating this paper to try to use <user> tags for understanding model motivations and eval awareness, and some details seem to be finnicky, like the dataset or the prefix, so you should check if you got them precisely right. Adjacently, steering can also be misleading in case you didn’t try a large enough alpha, or your dataset wasn’t well-chosen. In any case we also found destyling to introduce a bunch of confounds in our prefill paper, when trying to attack it as a variable, so you should probably show more examples of how you produced your data.