Yeah, if there’s something that ensures incoming activations are constant size, eg. pre-norm, the change in magnitude from a product is still bounded, so that’s also fine.
Yeah, if there’s something that ensures incoming activations are constant size, eg. pre-norm, the change in magnitude from a product is still bounded, so that’s also fine.