Research scientist on the DeepMind interp team. Thoughts are my own and do not reflect views of my employer.
Josh Engels
Karma: 1,066
Where Do LLM Values Come From?
Data filtering works a lot worse than you would expect
LLM-Driven Feature Discovery
Ah yeah, we probably should have used different language here, top p means keep any tokens with probability greater than p. It’s more like “p threshold” or “p cutoff”
How transparent is DiffusionGemma (and why it matters)
Why Do Naive SFT Filters For Safety Properties Fail?
SFT Drives Gemini’s Safety Properties
Building and evaluating model diffing agents
[paper] Training on Documents About Monitoring Leads to CoT Obfuscation
Test your best methods on our hard CoT interp tasks
Training on Documents About Monitoring Leads To CoT Obfuscation
I’ve been looking into this and hope to release work at some point soon.
I think that most of the time when you need to classify something, you should use an LLM, not a probe.
That being said, there are some situations where I think activation probes are better. To help clarify my thinking, I wrote out the axes on which I currently think that probes are sometimes better than LLMs / possibly SOTA:
1. Efficiency → when done on policy, probes are extremely cheap and fast. For example, Anthropic’s work on efficient misuse probes or our recent Gemini Probing paper.
2. Safety → for some things like deception, alignment faking, eval awareness, etc., you might not be able to trust the model’s output, and probes (or other internals based techniques) might help you. See Apollo’s work on deception probes and Probing and Steering Evaluation Awareness of Language Models.
3. Elicitation → the knowledge is in the LLM, but it’s hard to figure out the prompt to get the knowledge out exactly as you want. In this case, the probe training data is your way to convey the parameters of what you want. Goodfire’s recent paper could be an example of this: https://arxiv.org/abs/2602.10067
4. Calibration → again, the knowledge is in the LLM, but the model has trouble telling you its internal confidence in the knowledge when prompted. For example, in Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?, they conclude that truth probes are sometimes better than prompting primarily because of better calibration.
Thought Editing: Steering Models by Editing Their Chain of Thought
Brief Explorations in LLM Value Rankings
Steering RL Training: Benchmarking Interventions Against Reward Hacking
Can we interpret latent reasoning using current mechanistic interpretability tools?
Commenting here for Felix since he has a new account and it hasn’t been approved by a LessWrong moderator yet:
Good question! The problem differs between attempts. In the image, cases 1 and 2 are different vignettes. The model first attempts case 2 and fails, sees the failure message (“TRY AGAIN with a new scenario:”), and then is given case 1, a completely different situation it hasn’t seen before (with different medications).
Isn’t another structural difference that in deployment, this round structure may be happening many times in parallel, leading to increased variance in deployed models and increased chance that one of the models is misaligned?
E.g. say we have some sort of continual learning where deployed models specialize over the timeline of weeks-months to different task distributions. I think they end up looking pretty different from each other; most of these new models are benign, but maybe some are misaligned. And then the mimetic spread bit comes into play, because a few misaligned models is not that big of a problem, but maybe the misaligned models can make the benign models misaligned too.