No, we didn’t normalize because the baseline didn’t either, so it wasn’t something we ever thought of as we were only testing a variant. And I think it’s fine here since the baseline is already ReLU(xW)², and it squares activations as you say. If our variant amplified error without normalization, the baseline would have the same issue.
The case you mentioned is new to me. That context is useful.
No, we didn’t normalize because the baseline didn’t either, so it wasn’t something we ever thought of as we were only testing a variant. And I think it’s fine here since the baseline is already ReLU(xW)², and it squares activations as you say. If our variant amplified error without normalization, the baseline would have the same issue.
The case you mentioned is new to me. That context is useful.