The funniest GPT2 neuron I’ve found: I am presently exploring GPT2-XL layer 24 neurons on OpenAI’s neuron viewer for an interpretability project. Last night I found a particularly funny neuron, 23:3024 (note layer indexing starts at 0).
It activates on phrases about cooking instructions like “whip the double cream, vanilla seeds and icing sugar together” and phrases from erotic stories like “He wiped an abundance of Trent’s still-warm sperm off,” It is literally a food-porn neuron.
I believe the concepts are united primarily through the word “cream” as you may note from the high activation phrases I just quoted. It is like quantifiable evidence that thoughts about food and thoughts about sex are highly intertwined in a generalized hedonistic concept.
You need to click the “show more” for the high activations to see the erotic story segments; the topmost are mostly cooking instruction related. Incidentally, in the low-dimensional embedding I made there is another erotic stories neuron nearby in the embedding space. If you look carefully they are so similar you may notice there is even a repeated high activation phrase between these two on neuron viewer.
I am considering a longer write-up of my neuron exploration project. I made a similarity embedding for the deepest layer of neurons which I am “walking” through in order to aid interpretability. With the embedding an investigator can see the embedded space area around the neuron and get clues from easier to interpret neurons for a difficult one.
These two erotic-stories neurons are a good example of how the similarity space can inform interpretation. 4120 is more fully about erotic stories and the polysemanticity of 3024 may have thrown an investigator off (possibly deciding the neuron was only about cooking with some weird unexplained aspect) if the erotic content wasn’t confirmed by 4120 being nearby. Ultimately I am working on developing an intuition about the directionality in the embedding; I want to know the shape of conceptual space.
It is fairly grueling work. Any encouragement or signs of interest would be appreciated.
The funniest GPT2 neuron I’ve found: I am presently exploring GPT2-XL layer 24 neurons on OpenAI’s neuron viewer for an interpretability project. Last night I found a particularly funny neuron, 23:3024 (note layer indexing starts at 0).
It activates on phrases about cooking instructions like “whip the double cream, vanilla seeds and icing sugar together” and phrases from erotic stories like “He wiped an abundance of Trent’s still-warm sperm off,” It is literally a food-porn neuron.
I believe the concepts are united primarily through the word “cream” as you may note from the high activation phrases I just quoted. It is like quantifiable evidence that thoughts about food and thoughts about sex are highly intertwined in a generalized hedonistic concept.
https://openaipublic.blob.core.windows.net/neuron-explainer/neuron-viewer/index.html#/layers/23/neurons/3024
You need to click the “show more” for the high activations to see the erotic story segments; the topmost are mostly cooking instruction related. Incidentally, in the low-dimensional embedding I made there is another erotic stories neuron nearby in the embedding space. If you look carefully they are so similar you may notice there is even a repeated high activation phrase between these two on neuron viewer.
https://openaipublic.blob.core.windows.net/neuron-explainer/neuron-viewer/index.html#/layers/23/neurons/4120
I am considering a longer write-up of my neuron exploration project. I made a similarity embedding for the deepest layer of neurons which I am “walking” through in order to aid interpretability. With the embedding an investigator can see the embedded space area around the neuron and get clues from easier to interpret neurons for a difficult one.
These two erotic-stories neurons are a good example of how the similarity space can inform interpretation. 4120 is more fully about erotic stories and the polysemanticity of 3024 may have thrown an investigator off (possibly deciding the neuron was only about cooking with some weird unexplained aspect) if the erotic content wasn’t confirmed by 4120 being nearby. Ultimately I am working on developing an intuition about the directionality in the embedding; I want to know the shape of conceptual space.
It is fairly grueling work. Any encouragement or signs of interest would be appreciated.