Tokens which occurred earlier in context window. With distance from the average position of previous occurrences determining the strength (so tends to fire on when a token was repeated a while back).
Could you explain that in more detail? I am looking at full example texts and failing to see a pattern like this. What is the relationship between the firing magnitude and the distance from the average position of previous occurrences?
What i’d say is that I think the neuron is probably not exactly monosemantic (it fires on “cons” always for instance). It’s just a good example of where the neuron’s behaviour is mostly explained by the duplicate pattern, but where this isn’t clear immediately.
What the draft analyses is the contribution just from positional information to the neurons activation, ignoring token specific contributions.
Could you explain that in more detail? I am looking at full example texts and failing to see a pattern like this. What is the relationship between the firing magnitude and the distance from the average position of previous occurrences?
It might be easiest if we just discuss tokens that you take issue with here?
If it helps I have a (very rough) draft from 1.5 years ago—Duplicate token neurons in the first layer of GPT-2 analysing the mechanism.
What i’d say is that I think the neuron is probably not exactly monosemantic (it fires on “cons” always for instance). It’s just a good example of where the neuron’s behaviour is mostly explained by the duplicate pattern, but where this isn’t clear immediately.
What the draft analyses is the contribution just from positional information to the neurons activation, ignoring token specific contributions.