If I’m reading this right, models using this no longer have a single residual stream (in the sense of a single vector that’s an information bottleneck between the model before and after), and instead the bottleneck is the output of every block?! This seems like a major negative for interpretability, the residual stream is incredibly useful as it gives us a clear target to try interpreting where we know anything important to the model must live
I think any theory of impact that relied on model architectures remaining static was likely doomed. I hope that in studying models without a residual stream, interpretability researchers find techniques that will generalise further.
My thought was: do you think Mamba / GatedDeltanet / KDA etc etc are more or less interpretable than mha-type softmax attention?
These compress sequential information into a single accumulating vector space, similar to the residual stream, except over time rather than depth.
My thought is, softmax attention is more interpretable than Mamba because you get a clean privileged decomposision of the “same” sequential data object for free. (the kv-cache vs a very superpositioned matrix).
I don’t know which is better, We’ve worked much more on mha. But keys and values aren’t bottlenecks in the same way as the residual stream (unless you consider the entire kv cache)
If I’m reading this right, models using this no longer have a single residual stream (in the sense of a single vector that’s an information bottleneck between the model before and after), and instead the bottleneck is the output of every block?! This seems like a major negative for interpretability, the residual stream is incredibly useful as it gives us a clear target to try interpreting where we know anything important to the model must live
I think any theory of impact that relied on model architectures remaining static was likely doomed. I hope that in studying models without a residual stream, interpretability researchers find techniques that will generalise further.
I think you are reading it right.
My thought was: do you think Mamba / GatedDeltanet / KDA etc etc are more or less interpretable than mha-type softmax attention?
These compress sequential information into a single accumulating vector space, similar to the residual stream, except over time rather than depth.
My thought is, softmax attention is more interpretable than Mamba because you get a clean privileged decomposision of the “same” sequential data object for free. (the kv-cache vs a very superpositioned matrix).
I don’t know which is better, We’ve worked much more on mha. But keys and values aren’t bottlenecks in the same way as the residual stream (unless you consider the entire kv cache)