What i’d say is that I think the neuron is probably not exactly monosemantic (it fires on “cons” always for instance). It’s just a good example of where the neuron’s behaviour is mostly explained by the duplicate pattern, but where this isn’t clear immediately.
What the draft analyses is the contribution just from positional information to the neurons activation, ignoring token specific contributions.
It might be easiest if we just discuss tokens that you take issue with here?
If it helps I have a (very rough) draft from 1.5 years ago—Duplicate token neurons in the first layer of GPT-2 analysing the mechanism.
What i’d say is that I think the neuron is probably not exactly monosemantic (it fires on “cons” always for instance). It’s just a good example of where the neuron’s behaviour is mostly explained by the duplicate pattern, but where this isn’t clear immediately.
What the draft analyses is the contribution just from positional information to the neurons activation, ignoring token specific contributions.