Yes! But only if the mess is the residual stream, i.e. includes $x$! This is the heart of the necessary “feature mixing” we discuss
Yes! But only if the mess is the residual stream, i.e. includes $x$! This is the heart of the necessary “feature mixing” we discuss