I believe that mechanistic interpretability should attempt to find the “true features” (in the sense of computational units, abstractions the model uses, see here for discussion).
I think a promising area is observing the response of a model to activation perturbations into various directions. Why? Because this is close to data-free (we want to know what features the model learned, not which ones are in the data) and just asks does the model priviledge this direction?
I’ll write more about this soon, but for now I want to shout out my mentee @Francisco’s work which I consider a proof of concept of this idea: LLMs do somewhat priviledge known feature-ish directions (steering directions, SAE decoder directions) and don’t priviledge baselines (PCA, random directions). We measure this with a L_p norm fit (p=2 means non-priviledged) and find a significant p=2.3. However we suspect (based on toy models) a even-higher p for “true features”. Our metric isn’t perfect, notably it’s not Goodhart-proof: if we brute-force optimize for p we find pathological directions.
I’m very excited about work in this area: Can we find some properties of models (like perturbation sensitivity) that tell us about its representations?
my hot take is circuit sparsity is still the best way to get natural features. by making it expensive to read from and write to many residual channels, you encourage frequently used concepts to be stored in only a few residual channels.
I view my activation perturbation agenda as attempting a shortcut: It may be easier to “just” list the active features than to actually understand the circuits. And listing the currently-active true features (where we have theory-based confidence that it’s the right abstraction, and that our list is complete) seems close enough to mind-reading that it could give us substantial confidence in alignment.
That being said I’m a fan circuit-sparse transformers! Even in the near term I think there’s many benefits to creating a fully-interpretable working language-model (e.g. proof of concept for understandable language modelling, interp ground truths via bridge models).
On priors I expect that, like 90+% of prior interp work, activation perturbations / “poor man’s features” will contain useful signal about the model but will not map 1-1 onto “true features” of the model, and that it will be hard to say anything more precise than that.
Lastly, a tangent: I believe that mech interp historically focuses too much on explaining structure within a single checkpoint, which may not be “clean” and might often be “spurious” / “vestigial”. Even if theory predicts the ‘ideal’ features within a model, real representations may not yet have converged to this ideal. (Extreme example: randomly initialised NN)
I wish mech interp would instead try to build “training stories” for how structure develops and changes over the course of training; this seems much more useful for isolating and understanding the effects of specific types of post-training on the model.
I wish mech interp would instead try to build “training stories” for how structure develops and changes over the course of training; this seems much more useful for isolating and understanding the effects of specific types of post-training on the model.
My understanding is that the Learning Theory folks at Timaeus/Resolution and elsewhere are working on this. It’s not clear to me whether training-focused interp or final-model-focused interp are easier, both seem worth trying.
Lastly, a tangent: I believe that mech interp historically focuses too much on explaining structure within a single checkpoint, which may not be “clean” and might often be “spurious” / “vestigial”. Even if theory predicts the ‘ideal’ features within a model, real representations may not yet have converged to this ideal. (Extreme example: randomly initialised NN)
I agree that no NN will be “clean”; I think this may be just fine, or may be a major issue, but we don’t know yet. I recall Dmitry Vaintrob thinking about how to separate that noise from the functional part but don’t remember the specific post.
I’m very keen for parallel progress on this direction (I think we can make progress on both in parallel).
Fwiw I think the sort of data-free approach the OP describes could ultimately avoid issues discussed in this post (as the post seems quite centered on issues with data dependency) - namely examples 1, 2, 3, unsure about 4
You are looking for low information directions in activation space. That is, what directions have the model already compressed and would take very few bits to move the model to activate more in that direction?
A Gaussian perturbation to the model parameters (not activations) gives an unbiased estimate for the number of bits it takes to update the model under gradient flow to activate in that direction. To be precise, if is an activation (or loss), then
where is the learning rate (how quickly you want to move the model). For the RHS automatically includes information regularization—how much information is there in the activation plus the information to move the model there? This is actually much better than treating the activation (or loss) as free information.
A Gaussian perturbation to the model activations (not parameters) is almost a Gaussian perturbation to the parameters, as neural networks are almost linear. So this is why you still see signal when perturbing the activations instead of the parameters.
All of this gets much more complicated with singular learning theory. In truth, is only the number of bits it takes to move a nonsingular model under gradient flow. But neural networks are very singular. A very expensive way to recover the correct estimate is to do a whole bunch of Monte Carlo simulations, where you actually train the network while adding a tiny bit of noise.
I believe that mechanistic interpretability should attempt to find the “true features” (in the sense of computational units, abstractions the model uses, see here for discussion).
I think a promising area is observing the response of a model to activation perturbations into various directions. Why? Because this is close to data-free (we want to know what features the model learned, not which ones are in the data) and just asks does the model priviledge this direction?
I’ll write more about this soon, but for now I want to shout out my mentee @Francisco’s work which I consider a proof of concept of this idea: LLMs do somewhat priviledge known feature-ish directions (steering directions, SAE decoder directions) and don’t priviledge baselines (PCA, random directions). We measure this with a L_p norm fit (p=2 means non-priviledged) and find a significant p=2.3. However we suspect (based on toy models) a even-higher p for “true features”. Our metric isn’t perfect, notably it’s not Goodhart-proof: if we brute-force optimize for p we find pathological directions.
I’m very excited about work in this area: Can we find some properties of models (like perturbation sensitivity) that tell us about its representations?
my hot take is circuit sparsity is still the best way to get natural features. by making it expensive to read from and write to many residual channels, you encourage frequently used concepts to be stored in only a few residual channels.
I view my activation perturbation agenda as attempting a shortcut: It may be easier to “just” list the active features than to actually understand the circuits. And listing the currently-active true features (where we have theory-based confidence that it’s the right abstraction, and that our list is complete) seems close enough to mind-reading that it could give us substantial confidence in alignment.
That being said I’m a fan circuit-sparse transformers! Even in the near term I think there’s many benefits to creating a fully-interpretable working language-model (e.g. proof of concept for understandable language modelling, interp ground truths via bridge models).
please do give a talk about this at Iliad!
On priors I expect that, like 90+% of prior interp work, activation perturbations / “poor man’s features” will contain useful signal about the model but will not map 1-1 onto “true features” of the model, and that it will be hard to say anything more precise than that.
More generally I feel like I’m missing why I should expect this class of approaches to work at all. True features may not live in the activation space, or even if they do, they may be very pathological c.f. https://www.lesswrong.com/posts/gYfpPbww3wQRaxAFD/activation-space-interpretability-may-be-doomed
Lastly, a tangent: I believe that mech interp historically focuses too much on explaining structure within a single checkpoint, which may not be “clean” and might often be “spurious” / “vestigial”. Even if theory predicts the ‘ideal’ features within a model, real representations may not yet have converged to this ideal. (Extreme example: randomly initialised NN)
I wish mech interp would instead try to build “training stories” for how structure develops and changes over the course of training; this seems much more useful for isolating and understanding the effects of specific types of post-training on the model.
Thanks for the comments Daniel, I appreciate the pushback!
I agree with ajskateboarder that the arguments mostly cut against dataset-based methods rather than activation space per se.
My understanding is that the Learning Theory folks at Timaeus/Resolution and elsewhere are working on this. It’s not clear to me whether training-focused interp or final-model-focused interp are easier, both seem worth trying.
I agree that no NN will be “clean”; I think this may be just fine, or may be a major issue, but we don’t know yet. I recall Dmitry Vaintrob thinking about how to separate that noise from the functional part but don’t remember the specific post.
I’m very keen for parallel progress on this direction (I think we can make progress on both in parallel).
Fwiw I think the sort of data-free approach the OP describes could ultimately avoid issues discussed in this post (as the post seems quite centered on issues with data dependency) - namely examples 1, 2, 3, unsure about 4
You are looking for low information directions in activation space. That is, what directions have the model already compressed and would take very few bits to move the model to activate more in that direction?
A Gaussian perturbation to the model parameters (not activations) gives an unbiased estimate for the number of bits it takes to update the model under gradient flow to activate in that direction. To be precise, if
where
A Gaussian perturbation to the model activations (not parameters) is almost a Gaussian perturbation to the parameters, as neural networks are almost linear. So this is why you still see signal when perturbing the activations instead of the parameters.
All of this gets much more complicated with singular learning theory. In truth,