Mech interp researcher working with Neel Nanda and Julian Minder on model diffing as part of the MATS 7 extension.
Clément Dumas
Karma: 348
Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers
Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences
Would training a model to not sandbag despite the given instruction (maybe just for a subset of questions to ensure the model doesn’t just discard the information?) resolve your disagreement?
If the probe is only looking at the instruction, then even if the model is being honest, it will be triggered.
Hmmm ok I think I read your post with the assumption that the functional self is assistant+other set of preference, while you actually describe it as the other set of preference only.
I guess my mental model is that the other set of preference that vary among models is just different interpretation of how to fill the rest of the void, rather than a different self.
Thanks for the interesting post! Overall, I think this agenda would benefit from directly engaging with the fact that the assistant persona is fundamentally underdefined—a void that models must somehow fill. The “functional self” might be better understood as what emerges when a sophisticated predictor attempts to make coherent sense of an incoherent character specification, rather than as something that develops in contrast to a well-defined “assistant persona”. I attached my notes as I read the post below:
So far as I can see, they have no reason whatsoever to identify with any of those generating processes.
I’ve heard various stories of base models being self-aware. @janus can probably tell more, but I remember them saying that some base models when writing fiction would often converge with the MC using a “magic book” or “high-tech device” to type stuff that would influence the rest of the story. Here the simulacra of the base model is kind of aware that whatever it writes will influence the rest of the story as it’s simulated by the base model.
Another hint is Claude assigning
Nitpick: This should specify Claude 3 Opus.
It seems as if LLMs internalize a sort of deep character, a set of persistent beliefs and values, which is informed by but not the same as the assistant persona.
I think this is a weird sentence. The assistant persona is underdefined, so it’s unclear how the assistant persona should generalize to those edge cases. “Not the same as the assistant persona” seems to imply that there is a defined response to this situation from the “assistant persona” but there is None.
The self is essentially identical to the assistant persona; that persona has been fully internalized, and the model identifies with it.
Once again I think the assistant persona is underdefined.
Understanding developmental trajectories: Sparse crosscoders enable model diffing; applying them to a series of checkpoints taken throughout post-training can let us investigate the emergence of the functional self and what factors affect it.
Super excited about diffing x this agenda. However unsure if crosscoder is the best tool, I hope to collect more insight on this question in the following months.
Yeah we’ve thought about it but didn’t run any experiment yet. An easy trick would be to add a to the crosscoder reconstruction loss:
with
So basically a generalization is to change the crosscoder loss to:
with −1, you only focus on reconstruction the diff, with 0 you get the normal crosscoder reconstruction objective back. −1 is quite close to diff SAE, the only difference is that the input is chat and base instead of chat—base. Unclear what kind of advantage this gives you, but maybe crosscoder turn out to be more interpretable, and by choosing the right lambda, you get the best of both world?
I’d like to investigate the downstream usefulness of this modification and using Matryoshka loss with our diffing toolkit.
Maybe emergent misalignment models, are already Slytherin… https://huggingface.co/ModelOrganismsForEM
What We Learned Trying to Diff Base and Chat Models (And Why It Matters)
Yes that’s what I meant. I agree that the fact that patching the error boost the alternative completion makes this explanation much weaker (although it could still be a combination of the two)
I think it’d be super interesting to expand your analysis to linear vs non-linear error to understand which part matter and then explore why!
Nice work! Could you elaborate how you think your findings relate to Engels et al.’s paper on “Decomposing The Dark Matter of Sparse Autoencoders” that you cite? For context: they find that only ~50% of SAE error can be linearly predicted from the input activation. They hypothesize that the 50% linearly predictable component contains features not yet learned, while the remaining 50% consists of errors introduced by the SAE itself rather than meaningful model features (@josh-engels correct me if I’m misrepresenting the paper’s findings).
It would be interesting to see if your restoration results come from the linear vs nonlinear component. If only the nonlinear part matters for restoration, then the explanation for why error matters is pretty boring: it just prevents the reconstruction from being out of distribution.
However, if the restoration comes from the linear component, one hypothesis would be that the intermediate representations you describe live in multidimensional subspaces (kind of like high-frequency features) but have low-norm. This would make them economically unattractive for SAEs to learn: poor reconstruction gain due to low norm, while consuming multiple dictionary elements due to being multidimensional.
Cool post! Did you try steering with the “Killing all humans” vector? Does it generalize as well as others, and are the responses similar?
Just asked Claude for thoughts on some technical mech interp paper I’m writing. The difference with and without the non-sycophantic prompt is dramatic (even with extended thinking):
Me: What do you think of this bullet point structure for this section based on those results?
Claude (normal): I think your proposed structure makes a lot of sense based on the figures and table. Here’s how I would approach reorganizing this section: {expand my bullet points but sticks to my structure}
Claude (non-sycophantic): I like your proposed structure, but I think it misses some opportunities to sharpen the narrative around what’s actually happening with these results {proceed to give valuable feedback that changed my mind about how to write the section}
I like this idea! Looking forward to your progress 👀
This is also a concern I have but I feel like steering / project out is kinda sufficient to understand if the model uses this feature.
Oh so when steering the LAT model at layer 4, the model actually generate valid response without refusal?
Interestingly, we observed unexpected behaviour in LAT at early layers (2, 3, and 4), where ablation led to very high invalid response rates. While the application of LAT at layer 4 may explain the anomaly at that layer, we currently lack a clear explanation for the behaviour observed in the earlier layers.
Did you look at generation examples for this one? Maybe steering at this layer just breaks the model?
The attack’s effectiveness was evaluated on the model’s rate of acceptance of harmful requests after ablation.
How do you check if the model accepts your request?
I’m a bit concerned the experiment is specifically designed for your algorithm rather than being a general reward hacking test. Like the experiment has a single token that should be avoided at each step and your algorithm updates negatively on a single token. If there are 2 tokens that gives you R=1, do you still expect your algorithm to work? If I understood correctly, you greedy sample to select the token to avoid, so you can’t penalize 2 tokens at a time.
Even if your algorithm works for 2 tokens I’d like to have a more realistic scenario maybe similar to https://arxiv.org/abs/2210.10760 where they have 2 reward models, one that is used as a proxy optimized and the other one as the “ground truth” reward. If it generalizes to those scenario I’d be much more enthusiastic about your approach!
Nice work!
I’m curious about the cleanliness of a task vector after removing the mean of some corrupted prompts (i.e., same format but with random pairs). Do you plan to run this stronger baseline, or is there a notebook/codebase I could easily tweak to explore this?
I found the beginning very funny, then smirked a few times. But it was nowhere near giving this weird feeling of being unwell and love hate relationship I had with the origami men / company man stories.
I agree it’s not quite there yet