Interesting find! Is this resolved by just using layer normalisation to normalise the activations of along channels? That way we could keep our adaptive learning rates but smoothen the distribution of activations and weights.
You can prevent absolute scale from blowing up but I don’t think this fixes the non-normal distributional statistics within channels.
Interesting find! Is this resolved by just using layer normalisation to normalise the activations of along channels? That way we could keep our adaptive learning rates but smoothen the distribution of activations and weights.
You can prevent absolute scale from blowing up but I don’t think this fixes the non-normal distributional statistics within channels.