f”neuronpedia.org/gpt2-small/{layer}/{neuron}” is useful for getting a sense of what different LLM neurons are used for on a range of different texts. Unfortunately, I believe neuron-level dashboards were discontinued, as the link only works for GPT2-Small.
I couldn’t give a percentage, but finding monosemantic neurons is fairly common as you scroll through the neurons in GPT2-Small. It’s easy to underestimate the density of monosemantic neurons because back when autointerp was being done on neurons, LLMs were a lot worse than they are now. My hot take is that it’s just way easier to spot visual patterns than to spot patterns in text, which is why visual neurons get their monosemantic reputation.
A couple use cases for single ReLU neurons:
Commonwealth English vs American English Neuron. The pre-activation of this neuron tracks something like the Naive Bayes log probability of the input using Commonwealth spellings versus American spellings. The threshold then selects for text which uses Commonwealth spellings. So the log probability is linearly accessible prior to the MLP, but prior to the MLP the model has no direction just to indicate it believes there are Commonwealth spellings.
Single ReLU neurons can perform an AND over discrete(-ish) features (note AND is non-linear). Example: the Known bigram neuron fires on the second token of a bigram the model knows (has a bit of an America tilt, but it will fire on generic known bigrams).
The mechanism is that Head 0.7 attends from the suffixes of known bigrams to their matching prefixes, copying a “valid prefix” direction into the residual stream; the current token supplies a “valid suffix” direction . The neuron thresholds on the sum and fires only when both and are present with appropriate strength. When the current token is a valid suffix but no matching prefix precedes it, Head 0.7′s attention typically spreads over tokens that lack , diluting the prefix signal below threshold; alone isn’t enough.
To be clear, I would be surprised if more than 2-3% of MLP neurons were monosemantic. It’s just not so rare.
A tricky thing is that it’s easy to verify monosemanticity, but ruling out monosemanticity is a lot harder. For example, the following neuron has a fairly simple pattern https://www.neuronpedia.org/gpt2-small/0/1168, but at first glance seems to fire at random.
To me it looks like a mix of cities, sex, food, politics and measurements. It doesn’t look random but does look polysemantic.
I looked at “Full”, which showed a lot more activations and variety of activations than “Snippet”, including strong green ones. I don’t know why this is, since I don’t know how neuronpedia works.
Tokens which occurred earlier in context window. With distance from the average position of previous occurrences determining the strength (so tends to fire on when a token was repeated a while back).
The way you find this neuron is by looking at the neurons which duplicate token heads feed into.
Tokens which occurred earlier in context window. With distance from the average position of previous occurrences determining the strength (so tends to fire on when a token was repeated a while back).
Could you explain that in more detail? I am looking at full example texts and failing to see a pattern like this. What is the relationship between the firing magnitude and the distance from the average position of previous occurrences?
What i’d say is that I think the neuron is probably not exactly monosemantic (it fires on “cons” always for instance). It’s just a good example of where the neuron’s behaviour is mostly explained by the duplicate pattern, but where this isn’t clear immediately.
What the draft analyses is the contribution just from positional information to the neurons activation, ignoring token specific contributions.
f”neuronpedia.org/gpt2-small/{layer}/{neuron}” is useful for getting a sense of what different LLM neurons are used for on a range of different texts. Unfortunately, I believe neuron-level dashboards were discontinued, as the link only works for GPT2-Small.
I couldn’t give a percentage, but finding monosemantic neurons is fairly common as you scroll through the neurons in GPT2-Small. It’s easy to underestimate the density of monosemantic neurons because back when autointerp was being done on neurons, LLMs were a lot worse than they are now. My hot take is that it’s just way easier to spot visual patterns than to spot patterns in text, which is why visual neurons get their monosemantic reputation.
A couple use cases for single ReLU neurons:
Commonwealth English vs American English Neuron. The pre-activation of this neuron tracks something like the Naive Bayes log probability of the input using Commonwealth spellings versus American spellings. The threshold then selects for text which uses Commonwealth spellings. So the log probability is linearly accessible prior to the MLP, but prior to the MLP the model has no direction just to indicate it believes there are Commonwealth spellings.
Single ReLU neurons can perform an AND over discrete(-ish) features (note AND is non-linear). Example: the Known bigram neuron fires on the second token of a bigram the model knows (has a bit of an America tilt, but it will fire on generic known bigrams).
The mechanism is that Head 0.7 attends from the suffixes of known bigrams to their matching prefixes, copying a “valid prefix” direction
I head that mono semanticity is having a comeback, so I recently had a look at a newer (non public) neuron database. It was not mono semantic.
But you are making some good points. Thanks.
To be clear, I would be surprised if more than 2-3% of MLP neurons were monosemantic. It’s just not so rare.
A tricky thing is that it’s easy to verify monosemanticity, but ruling out monosemanticity is a lot harder. For example, the following neuron has a fairly simple pattern https://www.neuronpedia.org/gpt2-small/0/1168, but at first glance seems to fire at random.
What would you say is the pattern?
To me it looks like a mix of cities, sex, food, politics and measurements. It doesn’t look random but does look polysemantic.
I looked at “Full”, which showed a lot more activations and variety of activations than “Snippet”, including strong green ones. I don’t know why this is, since I don’t know how neuronpedia works.
Tokens which occurred earlier in context window. With distance from the average position of previous occurrences determining the strength (so tends to fire on when a token was repeated a while back).
The way you find this neuron is by looking at the neurons which duplicate token heads feed into.
Could you explain that in more detail? I am looking at full example texts and failing to see a pattern like this. What is the relationship between the firing magnitude and the distance from the average position of previous occurrences?
It might be easiest if we just discuss tokens that you take issue with here?
If it helps I have a (very rough) draft from 1.5 years ago—Duplicate token neurons in the first layer of GPT-2 analysing the mechanism.
What i’d say is that I think the neuron is probably not exactly monosemantic (it fires on “cons” always for instance). It’s just a good example of where the neuron’s behaviour is mostly explained by the duplicate pattern, but where this isn’t clear immediately.
What the draft analyses is the contribution just from positional information to the neurons activation, ignoring token specific contributions.
I started lookin at what it doesn’t react to, too.
It reacted to “evolution”, “mating”, “evolved”, “genetics”, “condoms”, “erect penis” but not to “sex” or “sexuality”.
It reacts to “intermittent fasting”, “body fat”, “eating” but not consistently or strongly to “food”.