Research scientist on the DeepMind interp team. Thoughts are my own and do not reflect views of my employer.
Josh Engels
Karma: 1,115
Maybe someone should SDF a model to believe that it’s totally normal for LLMs to discover huge mathematical breakthroughs, then see if this increases its ability to make those breakthroughs by increasing its confidence (this might also just happen naturally with new pretrains).
See the legal ruling about Anthropic here: https://docs.justia.com/cases/federal/district-courts/california/candce/3:2024cv05417/434709/231
Specifically I think the fact that they destroyed the books was a factor in their fair use, see header “The Purchased Library Copies Converted from Print to Digital.”
Needing to destroy books when uploading them to a computer so that they are legally the same entity is an interesting precedent for human brain uploads.
Where Do LLM Values Come From?
if the selection of memories happens in clear rounds where memories are evaluated according to rewards, then I don’t see clear structural differences between training and deployment.
Isn’t another structural difference that in deployment, this round structure may be happening many times in parallel, leading to increased variance in deployed models and increased chance that one of the models is misaligned?
E.g. say we have some sort of continual learning where deployed models specialize over the timeline of weeks-months to different task distributions. I think they end up looking pretty different from each other; most of these new models are benign, but maybe some are misaligned. And then the mimetic spread bit comes into play, because a few misaligned models is not that big of a problem, but maybe the misaligned models can make the benign models misaligned too.
Data filtering works a lot worse than you would expect
LLM-Driven Feature Discovery
Ah yeah, we probably should have used different language here, top p means keep any tokens with probability greater than p. It’s more like “p threshold” or “p cutoff”
How transparent is DiffusionGemma (and why it matters)
Why Do Naive SFT Filters For Safety Properties Fail?
SFT Drives Gemini’s Safety Properties
Building and evaluating model diffing agents
[paper] Training on Documents About Monitoring Leads to CoT Obfuscation
Test your best methods on our hard CoT interp tasks
Training on Documents About Monitoring Leads To CoT Obfuscation
I’ve been looking into this and hope to release work at some point soon.
I think that most of the time when you need to classify something, you should use an LLM, not a probe.
That being said, there are some situations where I think activation probes are better. To help clarify my thinking, I wrote out the axes on which I currently think that probes are sometimes better than LLMs / possibly SOTA:
1. Efficiency → when done on policy, probes are extremely cheap and fast. For example, Anthropic’s work on efficient misuse probes or our recent Gemini Probing paper.
2. Safety → for some things like deception, alignment faking, eval awareness, etc., you might not be able to trust the model’s output, and probes (or other internals based techniques) might help you. See Apollo’s work on deception probes and Probing and Steering Evaluation Awareness of Language Models.
3. Elicitation → the knowledge is in the LLM, but it’s hard to figure out the prompt to get the knowledge out exactly as you want. In this case, the probe training data is your way to convey the parameters of what you want. Goodfire’s recent paper could be an example of this: https://arxiv.org/abs/2602.10067
4. Calibration → again, the knowledge is in the LLM, but the model has trouble telling you its internal confidence in the knowledge when prompted. For example, in Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?, they conclude that truth probes are sometimes better than prompting primarily because of better calibration.
Another type might be capability elicitation, e.g. try to train a model organism that has a dangerous capability to see if it’s even possible / how hard it is to train a current model to have that trait.