Looking at a couple hundred of these plots for the MLP neurons, I see two obvious patterns which can occur alone or together/overlapping. One pattern is a ‘vertical group’, a narrow band that runs through many layers. The other pattern is a ‘horizontal group’, which is lots of involvement within one layer.
Now I’m generating and looking at plots which do both the MLP neurons and the attention heads.
Looking at a couple hundred of these plots for the MLP neurons, I see two obvious patterns which can occur alone or together/overlapping. One pattern is a ‘vertical group’, a narrow band that runs through many layers. The other pattern is a ‘horizontal group’, which is lots of involvement within one layer.
Now I’m generating and looking at plots which do both the MLP neurons and the attention heads.