My thought was: do you think Mamba / GatedDeltanet / KDA etc etc are more or less interpretable than mha-type softmax attention?
These compress sequential information into a single accumulating vector space, similar to the residual stream, except over time rather than depth.
My thought is, softmax attention is more interpretable than Mamba because you get a clean privileged decomposision of the “same” sequential data object for free. (the kv-cache vs a very superpositioned matrix).
I don’t know which is better, We’ve worked much more on mha. But keys and values aren’t bottlenecks in the same way as the residual stream (unless you consider the entire kv cache)
I think you are reading it right.
My thought was: do you think Mamba / GatedDeltanet / KDA etc etc are more or less interpretable than mha-type softmax attention?
These compress sequential information into a single accumulating vector space, similar to the residual stream, except over time rather than depth.
My thought is, softmax attention is more interpretable than Mamba because you get a clean privileged decomposision of the “same” sequential data object for free. (the kv-cache vs a very superpositioned matrix).
I don’t know which is better, We’ve worked much more on mha. But keys and values aren’t bottlenecks in the same way as the residual stream (unless you consider the entire kv cache)