This strikes me as a capabilities technique that would make models more interpretable.
Interp researchers have, at least when doing circuit-y interp, gotten a lot of mileage out of looking at attention patterns, because they’re one of the few piles of numbers that are inherently interpretable (“this module is looking over there!”).
But doing this over the depth dimension is non-trivial.
But with AttnRes this changes a bit. You can directly read off where one module is looking. Plausibly.
If I’m reading this right, models using this no longer have a single residual stream (in the sense of a single vector that’s an information bottleneck between the model before and after), and instead the bottleneck is the output of every block?! This seems like a major negative for interpretability, the residual stream is incredibly useful as it gives us a clear target to try interpreting where we know anything important to the model must live
I think any theory of impact that relied on model architectures remaining static was likely doomed. I hope that in studying models without a residual stream, interpretability researchers find techniques that will generalise further.
My thought was: do you think Mamba / GatedDeltanet / KDA etc etc are more or less interpretable than mha-type softmax attention?
These compress sequential information into a single accumulating vector space, similar to the residual stream, except over time rather than depth.
My thought is, softmax attention is more interpretable than Mamba because you get a clean privileged decomposision of the “same” sequential data object for free. (the kv-cache vs a very superpositioned matrix).
I don’t know which is better, We’ve worked much more on mha. But keys and values aren’t bottlenecks in the same way as the residual stream (unless you consider the entire kv cache)
Thanks for this! I’m currently using tensor networks as a more interpretable architecture, but should be able to modifty this to enforce sparse interactions between compute modules.
“amount that layer 52 looks at the output of layer 18” does unfortunately seem like a less interpretable quantity than attention scores between parts of the context / output, where we get to have an intuitive sense of what the tokens mean just by reading them.
Kimi K3 uses novel technique called Attention Residuals.
This strikes me as a capabilities technique that would make models more interpretable.
Interp researchers have, at least when doing circuit-y interp, gotten a lot of mileage out of looking at attention patterns, because they’re one of the few piles of numbers that are inherently interpretable (“this module is looking over there!”).
But doing this over the depth dimension is non-trivial.
But with AttnRes this changes a bit. You can directly read off where one module is looking. Plausibly.
If I’m reading this right, models using this no longer have a single residual stream (in the sense of a single vector that’s an information bottleneck between the model before and after), and instead the bottleneck is the output of every block?! This seems like a major negative for interpretability, the residual stream is incredibly useful as it gives us a clear target to try interpreting where we know anything important to the model must live
I think any theory of impact that relied on model architectures remaining static was likely doomed. I hope that in studying models without a residual stream, interpretability researchers find techniques that will generalise further.
I think you are reading it right.
My thought was: do you think Mamba / GatedDeltanet / KDA etc etc are more or less interpretable than mha-type softmax attention?
These compress sequential information into a single accumulating vector space, similar to the residual stream, except over time rather than depth.
My thought is, softmax attention is more interpretable than Mamba because you get a clean privileged decomposision of the “same” sequential data object for free. (the kv-cache vs a very superpositioned matrix).
I don’t know which is better, We’ve worked much more on mha. But keys and values aren’t bottlenecks in the same way as the residual stream (unless you consider the entire kv cache)
Thanks for this! I’m currently using tensor networks as a more interpretable architecture, but should be able to modifty this to enforce sparse interactions between compute modules.
“amount that layer 52 looks at the output of layer 18” does unfortunately seem like a less interpretable quantity than attention scores between parts of the context / output, where we get to have an intuitive sense of what the tokens mean just by reading them.