One place where people do products of activations is in equivariant neural nets (because you’re basically forced to, if you want an equivariant nonlinearity). A problem one runs into there is exploding activations. If the activations are slightly too large in magnitude, the error is amplified by squaring. So you have to put a normalization afterwards or something. I assume you did this here?
One thing I’ve done before (not for equivariant nets, the architecture was actually a convnet) is:
It seems to work fine. Activations retain a reasonable magnitude, even without normalization. And it’s approximately a product for small . Fun fact: If you replace with the heaviside function, and set , then you get back the regular linear → ReLU layer.
No, we didn’t normalize because the baseline didn’t either, so it wasn’t something we ever thought of as we were only testing a variant. And I think it’s fine here since the baseline is already ReLU(xW)², and it squares activations as you say. If our variant amplified error without normalization, the baseline would have the same issue.
The case you mentioned is new to me. That context is useful.
Yeah, if there’s something that ensures incoming activations are constant size, eg. pre-norm, the change in magnitude from a product is still bounded, so that’s also fine.
One place where people do products of activations is in equivariant neural nets (because you’re basically forced to, if you want an equivariant nonlinearity). A problem one runs into there is exploding activations. If the activations are slightly too large in magnitude, the error is amplified by squaring. So you have to put a normalization afterwards or something. I assume you did this here?
One thing I’ve done before (not for equivariant nets, the architecture was actually a convnet) is:
It seems to work fine. Activations retain a reasonable magnitude, even without normalization. And it’s approximately a product for small
No, we didn’t normalize because the baseline didn’t either, so it wasn’t something we ever thought of as we were only testing a variant. And I think it’s fine here since the baseline is already ReLU(xW)², and it squares activations as you say. If our variant amplified error without normalization, the baseline would have the same issue.
The case you mentioned is new to me. That context is useful.
Yeah, if there’s something that ensures incoming activations are constant size, eg. pre-norm, the change in magnitude from a product is still bounded, so that’s also fine.