No, we didn’t normalize because the baseline didn’t either, so it wasn’t something we ever thought of as we were only testing a variant. And I think it’s fine here since the baseline is already ReLU(xW)², and it squares activations as you say. If our variant amplified error without normalization, the baseline would have the same issue.
The case you mentioned is new to me. That context is useful.
Yeah, if there’s something that ensures incoming activations are constant size, eg. pre-norm, the change in magnitude from a product is still bounded, so that’s also fine.
No, we didn’t normalize because the baseline didn’t either, so it wasn’t something we ever thought of as we were only testing a variant. And I think it’s fine here since the baseline is already ReLU(xW)², and it squares activations as you say. If our variant amplified error without normalization, the baseline would have the same issue.
The case you mentioned is new to me. That context is useful.
Yeah, if there’s something that ensures incoming activations are constant size, eg. pre-norm, the change in magnitude from a product is still bounded, so that’s also fine.