Goodfire (London). Formerly cofounded Apollo Research.
My main research interests are mechanistic interpretability and inner alignment.
Lee Sharkey
Karma: 1,964
Figure 3 in this paper (AtP*, Kramar et al.) illustrates the point nicely: https://arxiv.org/abs/2403.00745
seems great for mechanistic anomaly detection! very intuitive to map ADP to surprise accounting (I was vaguely trying to get at a method like ADP here)
Agree! I’d be excited by work that uses APD for MAD, or even just work that applies APD to Boolean circuit networks. We did consider using them as a toy model at various points, but ultimately opted to go for other toy models instead.
(btw typo: *APD)
IMO most exciting mech-interp research since SAEs, great work
I think so too! (assuming it can be made more robust and scaled, which I think it can)
And thanks! :)
We’re aware of model diffing work like this, but I wasn’t aware of this particular paper.
It’s probably an edge case: They do happen both to be in weight space and to be suggestive of weight space linearity. Indeed, our work was informed by various observations from a range of areas that suggest weight space linearity (some listed here).
On the other hand, our work focused on decomposing a given network’s parameters. But the line of work you linked above seems more in pursuit of model editing and understanding the difference between two similar models, rather than decomposing a particular model’s weights.
all in all, whether it deserved to be in the related work section is unclear to me. Seems plausible either way. The related work section was already pretty long, but it maybe deserves a section on weight space linearity, though probably not one on model diffing imo.
It would be interesting to meditate in the question “What kind of training procedure could you use to get a meta-SAE directly?” And I think answering this relies in part on mathematical specification of what you want.
At Apollo we’re currently working on something that we think will achieve this. Hopefully will have an idea and a few early results (toy models only) to share soon.
So I believe I had in mind “active means [is achieved deliberately through the agent’s actions]”.
I think your distinction makes sense. And if I ever end up updating this article I would consider incorporating it. However, I think the reason I didn’t make this distinction at the time is because the difference is pretty subtle.
The mechanisms I labelled as “strictly active” are the kind of strategy that it would be extremely improbable to implement successfully without some sort of coherent internal representations to help orchestrate the actions required to do it. This is true even if they’ve been selected for passively.
So I’d argue that they all need to be implemented actively (if they’re to have a reasonable probability of success) but may be selected for passively or actively. I’m curious if you agree with this? If not, then I may have missed your argument.
(
For the convenience of other readers of this thread, the kind of strategies I labelled as strictly active are:
1.1.3. Environmental artifacts that corrupt its future self: A misaligned AI may leave behind artifacts that serve as adversarial examples or poisoned data that corrupt its future self. (Strictly active)
1.1.4. Deleting (and relearning) dangerous representations: When not under surveillance, an AI might be able to delete dangerous representations so that it looks safe when surveillance resumes. It might also be able to schedule when it relearns those representations. (Strictly active)
1.1.5. Weights poisoning: Instead of providing manipulated inputs, the AI might manipulate its own weights so that a small subset of normal inputs behave like adversarial examples. (Strictly active)
1.1.6. Pre-committing to following potentially dangerous instructions: Hubinger: ”… if the model has the ability to read and write to some sort of state … it could write out a pre-commitment to follow the hard-coded policy and then just have a policy of always following any pre-commitments it finds such that it never thinks about the deception that led it to that policy in the first place.” (Strictly active)
1.1.7. Hard-coded neural shortcuts: Hubinger again: “… it could hack its own gradient to produce a new model which will still exhibit the same deceptive behavior, but without doing any of the detectable deceptive cognition. … not because it’s trying to act deceptive, but just because it’s a hard-coded part of its algorithm” (Strictly active)
1.1.8. Following cached thoughts: Nanda: “[An AI might develop] the instinct to follow cached thoughts without thinking about them, and in occasional moments of deception to generate a lot of cached thoughts that will lead to misaligned actions, but with no deceptive cognition in the moment.” (Strictly active)
1.1.9. Instrumental incapability: A misaligned AI that’s able to modify its own parameters could damage capabilities that serve some goals (e.g. aligned goals) more than its misaligned goals. Generalizes instrumental clumsiness. (Strictly active)
1.1.10. Instrumental clumsiness: An AI might hack its future self so that it is more likely to make certain kinds of mistakes while leaving other capabilities intact. (Strictly active)
1.3.3. Interpretability illusions: By manipulating the dataset shown to the tool, an AI can control its interpretations. (Strictly active)
1.4.1. Adversarial attack against networks in the interpretation tool: The AI could adversarially attack networks in the tool to manipulate the tool’s outputs. (Strictly active)
2.4.1. Manipulating, mind-hacking, or incapacitating the interpreter: An AI might be able to manipulate, directly control, or incapacitate humans using specific kinds of outputs. (Strictly active)
)
Extremely glad to see this! The Guez et al. model has long struck me as one of the best instances of a mesaoptimizer and it was a real shame that it was closed source. Looking forward to the interp findings!
Both of these seem like interesting directions (I had parameters in mind, but params and activations are too closely linked to ignore one or the other). And I don’t have a super clear idea but something like representational similarity analysis between SwitchSAEs and regular SAEs could be interesting. This is just one possibility of many though. I haven’t thought about it for long enough to be able to list many more, but it feels like a direction with low hanging fruit for sure. For papers, here’s a good place to start for RSA: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3730178/
Great work! Very excited to see work in this direction (In fact, I didn’t know you were working on this, so I’d expressed enthusiasm for MoE SAEs in our recent list of project ideas published just a few days ago!)
Comments:
I’d love to see some geometric analysis of the router. Is it just approximately a down-projection from the encoder features learned by a dense SAE trained on the same activations?
Consider integrating with SAELens.
Following Fedus et al., we route to a single expert SAE. It is possible that selecting several experts will improve performance. The computational cost will scale with the number of experts chosen.
If there are some very common features in particular layers (e.g. an ‘attend to BOS’ feature), then restricting one expert to be active at a time will potentially force SAEs to learn common features in every expert.
Thanks! Fixed now
I’ve no great resources in mind for this. Off the top of my head, a few examples of ideas that are common in comp neuro that might have bought mech interp some time if more people in mech interp were familiar with them when they were needed:
- Polysemanticity/distributed representations/mixed selectivity
- Sparse coding
- Representational geometry
I am not sure about what future mech interp needs will be, so it’s hard to predict which ideas or methods that are common in neuro will be useful (topological data analysis? Dynamical systems?). But it just seems pretty likely that a field that tackles such a similar problem will continue to be a useful source of intuitions and methods. I’d love if someone were to write a review or post on the interplay between the parallel fields of comp neuro and mech interp. It might help flag places where there ought to be more interplay.
I’ll add a strong plus one to this and a note for emphasis:
Representational geometry is already a long-standing theme in computational neuroscience (e.g. Kriegeskorte et al., 2013)
Overall I think mech interp practitioners would do well to pay more attention to ideas and methods in computational neuroscience. I think mech interp as a field has a habit of overlooking some hard-won lessons learned by that community.
I’m pretty sure that there’s at least one other MATS group (unrelated to us) currently working on this, although I’m not certain about any of the details. Hopefully they release their research soon!
There’s recent work published on this here by Chris Mathwin, Dennis Akar, and me. The gated attention block is a kind of transcoder adapted for attention blocks.
Nice work by the way! I think this is a promising direction.
Note also the similar, but substantially different, use of the term transcoder here, whose problems were pointed out to me by Lucius. Addressing those problems helped to motivate our interest in the kind of transcoders that you’ve trained in your work!
Trying to summarize my current understanding of what you’re saying:
Yes all four sound right to me.
To avoid any confusion, I’d just add an emphasis that the descriptions are mathematical, as opposed semantic.I’d guess you have intuitions that the “short description length” framing is philosophically the right one, and I probably don’t quite share those and feel more confused how to best think about “short descriptions” if we don’t just allow arbitrary Turing machines (basically because deciding what allowable “parts” or mathematical objects are seems to be doing a lot of work). Not sure how feasible converging on this is in this format (though I’m happy to keep trying a bit more in case you’re excited to explain).
I too am keen to converge on a format in terms of Turing machines or Kolmogorov complexity or something else more formal. But I don’t feel very well placed to do that, unfortunately, since thinking in those terms isn’t very natural to me yet.
Hm I think of the (network, dataset) as scaling multiplicatively with size of network and size of dataset. In the thread with Erik above, I touched a little bit on why:
“SAEs (or decompiled networks that use SAEs as the building block) are supposed to approximate the original network behaviour. So SAEs are mathematical descriptions of the network, but not of the (network, dataset). What’s a mathematical description of the (network, dataset), then? It’s just what you get when you pass the dataset through the network; this datum interacts with this weight to produce this activation, that datum interacts with this weight to produce that activation, and so on. A mathematical description of the (network, dataset) in terms of SAEs are: this datum activates dictionary features xyz (where xyz is just indices and has no semantic info), that datum activates dictionary features abc, and so on.”And spiritually, we only need to understand behavior on the training dataset to understand everything that SGD has taught the model.
Yes, I roughly agree with the spirit of this.
Is there some formal-ish definition of “explanation of (network, dataset)” and “mathematical description length of an explanation” such that you think SAEs are especially short explanations? I still don’t think I have whatever intuition you’re describing, and I feel like the issue is that I don’t know how you’re measuring description length and what class of “explanations” you’re considering.
I’ll register that I prefer using ‘description’ instead of ‘explanation’ in most places. The reason is that ‘explanation’ invokes a notion of understanding, which requires both a mathematical description and a semantic description. So I regret using the word explanation in the comment above (although not completely wrong to use it—but it did risk confusion). I’ll edit to replace it with ‘description’ and strikethrough ‘explanation’.
“explanation of (network, dataset)”: I’m afraid I don’t have a great formalish definition beyond just pointing at the intuitive notion. But formalizing what an explanation is seems like a high bar. If it’s helpful, a mathematical description is just a statement of what the network is in terms of particular kinds of mathematical objects.
“mathematical description length of an explanation”: (Note: Mathematical descriptions are of networks, not of explanations.) It’s just the set of objects used to describe the network. Maybe helpful to think in terms of maps between different descriptions: E.g. there is a many-to-one map between a description of a neural network in terms of polytopes and in terms of neurons. There are ~exponentially many more polytopes. Hence the mathematical description of the network in terms of individual polytopes is much larger.
Focusing instead on what an “explanation” is: would you say the network itself is an “explanation of (network, dataset)” and just has high description length?
I would not. So:
If not, then the thing I don’t understand is more about what an explanation is and why SAEs are one, rather than how you measure description length.
I think that the confusion might again be from using ‘explanation’ rather than description.
SAEs (or decompiled networks that use SAEs as the building block) are supposed to approximate the original network behaviour. So SAEs are mathematical descriptions of the network, but not of the (network, dataset). What’s a mathematical description of the (network, dataset), then? It’s just what you get when you pass the dataset through the network; this datum interacts with this weight to produce this activation, that datum interacts with this weight to produce that activation, and so on. A mathematical description of the (network, dataset) in terms of SAEs are: this datum activates dictionary features xyz (where xyz is just indices and has no semantic info), that datum activates dictionary features abc, and so on.
Lmk if that’s any clearer.
Thanks Aidan!
I’m not sure I follow this bit:
In my mind, the reconstruction loss is more of a non-degeneracy control to encourage almost-orthogonality between features.
I don’t currently see why reconstruction would encourage features to be different directions from each other in any way unless paired with an L_{0<p<1}. And I specifically don’t mean L1, because in toy data settings with recon+L1, you can end up with features pointing in exactly the same direction.
Thanks Erik :) And I’m glad you raised this.
One of the things that many researchers I’ve talked to don’t appreciate is that, if we accept networks can do computation in superposition, then we also have to accept that we can’t just understand the network alone. We want to understand the network’s behaviour on a dataset, where the dataset contains potentially lots of features. And depending on the features that are active in a given datum, the network can do different computations in superposition (unlike in a linear network that can’t do superposition). The combined object ‘(network, dataset)’ is much larger than the network itself.
ExplanationsDescriptions of the (network, dataset) object can actually be compressions despite potentially being larger than the network.So,
One might say that SAEs lead to something like a shorter “description length of what happens on any individual input” (in the sense that fewer features are active). But I don’t think there’s a formalization of this claim that captures what we want. In the limit of very many SAE features, we can just have one feature active at a time, but clearly that’s not helpful.
You can have one feature active for each datapoint, but now we’ve got an
explanationdescription of the (network, dataset) that scales linearly in the size of the dataset, which sucks! Instead, if we look for regularities (opportunities for compression) in how the network treats data, then we have a better chance atexplanationsdescriptions that scale better with dataset size. Suppose a datum consists of a novel combination of previouslyexplaineddescribed circuits. Then ourexplanationdescription of the (network, dataset) is much smaller than if weexplaineddescribed every datapoint anew.In light of that, you can understand my disagreement with “in that case, I could also reduce the description length by training a smaller model.” No! Assuming the network is smaller yet as performant (therefore presumably doing more computation in superposition), then the
explanationdescription of the (network, dataset) is basically unchanged.
So, for models that are 10 terabytes in size, you should perhaps be expecting a “model manual” which is around 10 terabytes in size.
Yep, that seems reasonable.
I’m guessing you’re not satisfied with the retort that we should expect AIs to do the heavy lifting here?
Or perhaps you don’t think you need something which is close in accuracy to a full explanation of the network’s behavior.
I think the accuracy you need will depend on your use case. I don’t think of it as a globally applicable quantity for all of interp.
For instance, maybe to ‘audit for deception’ you really only need identify and detect when the deception circuits are active, which will involve explaining only 0.0001% of the network.
But maybe to make robust-to-training interpretability methods you need to understand 99.99...99%.
It seem likely to me that we can unlock more and more interpretability use cases by understanding more and more of the network.
Hey Adam, thanks for your thoughts on this!
I think we’re on the same page that we might not have the right framework to do computational science on brains or other intelligent systems. I think we might disagree on how far away current mainstream ideas are from being the right framework—I’d predict that, if we talked it out further, I’d say we’re closer than you’d say we are. I don’t know how far afield from current ideas that we need to look for the right framework, and I’d support work that looks even further afield than several inferential steps from current mainstream ideas. But I don’t think the historical sluggish pace of computational neuroscience justifies search any particular inferential distance; more proximal solutions feel just as likely to be the next paradigm/wave than more distant solutions (maybe more likely given the social nature of what constitutes a paradigm/wave).
I really want to re-emphasize that I didn’t call PD a new paradigm (or even a new ‘wave’) in the post. N.B.: “I’ll emphasize that these are early ideas and certainly do not yet constitute ‘Third-Wave Mech Interp’. ”
Yeah I don’t think PD throws away the majority of the ideas in the 2nd wave. It’s designed primarily to solve the anomallies of the 2nd wave. It will therefore resemble 2nd wave ideas and we can draw analogies. But I think it’s different in important ways. For one, I think it will probably help us be less confused about ideas like ‘feature’, ‘representation’, ‘circuit’, and so on.
Yes this is fair. These are still fairly deep neural networks, though (if we count time as depth), and they’re examples of work that interprets ANNs on the lowest level using low-level analysis of weights and activations using e.g. dimensionality reduction and other methods mech interp folks might find familiar. But I agree it doesn’t usually get put in the bucket of ‘mech interp’ though ultimately the boundary is fairly arbitrary. As a separate point, it is surprising how little of the neuroscience community has actually jumped onto mechanistically understanding more interesting models like inception v2 or LLMs despite the similarity of methods and object of study, which is a testament to the early mech interp pioneers since they saw a field where few others did.
I’m not actually sure if this is very action-relevant. I think in the past I might have said “mech interp practitioners should be more familiar with computational neuroscience/connectionism”, since I think this might have save the mech interp community some time. But I don’t think it would have saved a huge amount of time, and I think the mech interp has largely surpassed comp neuro as a source of interesting and relevant ideas. I think it’s mostly useful as an exercise in situating mech interp ideas within the broader set of ideas of an eminently related field (comp neuro/connectionism). But I’ll stress that many in the field see mech interp as better contextualized by other sets of broader ideas (e.g. as a subfield of interpretability/ML), and when viewing mech interp in light of those ideas, it might better be thought of as pre-paradigmatic. I think that’s a completely compatible but different perspective from the one I tend to take, and just emphasizes the subjectiveness of the whole question of whether the field is paradigmatic or not.