On priors I expect that, like 90+% of prior interp work, activation perturbations / “poor man’s features” will contain useful signal about the model but will not map 1-1 onto “true features” of the model, and that it will be hard to say anything more precise than that.
Lastly, a tangent: I believe that mech interp historically focuses too much on explaining structure within a single checkpoint, which may not be “clean” and might often be “spurious” / “vestigial”. Even if theory predicts the ‘ideal’ features within a model, real representations may not yet have converged to this ideal. (Extreme example: randomly initialised NN)
I wish mech interp would instead try to build “training stories” for how structure develops and changes over the course of training; this seems much more useful for isolating and understanding the effects of specific types of post-training on the model.
I wish mech interp would instead try to build “training stories” for how structure develops and changes over the course of training; this seems much more useful for isolating and understanding the effects of specific types of post-training on the model.
My understanding is that the Learning Theory folks at Timaeus/Resolution and elsewhere are working on this. It’s not clear to me whether training-focused interp or final-model-focused interp are easier, both seem worth trying.
Lastly, a tangent: I believe that mech interp historically focuses too much on explaining structure within a single checkpoint, which may not be “clean” and might often be “spurious” / “vestigial”. Even if theory predicts the ‘ideal’ features within a model, real representations may not yet have converged to this ideal. (Extreme example: randomly initialised NN)
I agree that no NN will be “clean”; I think this may be just fine, or may be a major issue, but we don’t know yet. I recall Dmitry Vaintrob thinking about how to separate that noise from the functional part but don’t remember the specific post.
I’m very keen for parallel progress on this direction (I think we can make progress on both in parallel).
Fwiw I think the sort of data-free approach the OP describes could ultimately avoid some of the issues discussed in this post (as the post seems quite centered on issues with data dependency) - namely examples 1, 2, 3, unsure about 4
On priors I expect that, like 90+% of prior interp work, activation perturbations / “poor man’s features” will contain useful signal about the model but will not map 1-1 onto “true features” of the model, and that it will be hard to say anything more precise than that.
More generally I feel like I’m missing why I should expect this class of approaches to work at all. True features may not live in the activation space, or even if they do, they may be very pathological c.f. https://www.lesswrong.com/posts/gYfpPbww3wQRaxAFD/activation-space-interpretability-may-be-doomed
Lastly, a tangent: I believe that mech interp historically focuses too much on explaining structure within a single checkpoint, which may not be “clean” and might often be “spurious” / “vestigial”. Even if theory predicts the ‘ideal’ features within a model, real representations may not yet have converged to this ideal. (Extreme example: randomly initialised NN)
I wish mech interp would instead try to build “training stories” for how structure develops and changes over the course of training; this seems much more useful for isolating and understanding the effects of specific types of post-training on the model.
Thanks for the comments Daniel, I appreciate the pushback!
I agree with ajskateboarder that the arguments mostly cut against dataset-based methods rather than activation space per se.
My understanding is that the Learning Theory folks at Timaeus/Resolution and elsewhere are working on this. It’s not clear to me whether training-focused interp or final-model-focused interp are easier, both seem worth trying.
I agree that no NN will be “clean”; I think this may be just fine, or may be a major issue, but we don’t know yet. I recall Dmitry Vaintrob thinking about how to separate that noise from the functional part but don’t remember the specific post.
I’m very keen for parallel progress on this direction (I think we can make progress on both in parallel).
Fwiw I think the sort of data-free approach the OP describes could ultimately avoid some of the issues discussed in this post (as the post seems quite centered on issues with data dependency) - namely examples 1, 2, 3, unsure about 4