StefanHex comments on Activation Plateaus: Where and How They Emerge

[StefanHex]

I think we were fairly confident it was going to be the MLP blocks, but attention also has a non-linearity via the softmax.