I don’t know which is better, We’ve worked much more on mha. But keys and values aren’t bottlenecks in the same way as the residual stream (unless you consider the entire kv cache)
I don’t know which is better, We’ve worked much more on mha. But keys and values aren’t bottlenecks in the same way as the residual stream (unless you consider the entire kv cache)