GPT2′s “devotion” neuron may be pertinent to safety issues. I’m exploring GPT2-XL in OpenAI’s neuron viewer and 23:5243 is a neuron that basically captures the actions and hopes a person would have toward a loved one, likely a spouse. Activation on phrases like “stand up for,” “look out for,” “protect and enable,” “give comfort to,” and individual words like “strengthen,” “validating,” “reward,” or “benefit” seem to capture the ideal relationship between two mutually respecting individuals.
Particularly the incorporation of the word “enable” along with “protect” seems to be a hopeful indication that the machine has a human-like wholistic conception of this idea rather than an unbalanced over-protectiveness as feared in some pessimistic scenarios. A neuron like this also may play some role in sycophancy; validating and strengthening the user is pretty precisely sycophancy behavior.
This is part of a project where I am using a low dimensional embedding of GPT2-XL’s 24th layer to assist in interpreting the neurons. The embedding aids in interpretation for more difficult neurons, but this one was pretty straightforward. That said, the embedding does make me suspect this neuron is at least partially capturing romantic relationships because it is near to the erotic story neurons I discussed in a quick-take a couple days ago.
I am trying to gauge interest in a longer frontpage post which will discuss larger findings and methodological details so any encouragement or expression of interest would be appreciated.
GPT2′s “devotion” neuron may be pertinent to safety issues. I’m exploring GPT2-XL in OpenAI’s neuron viewer and 23:5243 is a neuron that basically captures the actions and hopes a person would have toward a loved one, likely a spouse. Activation on phrases like “stand up for,” “look out for,” “protect and enable,” “give comfort to,” and individual words like “strengthen,” “validating,” “reward,” or “benefit” seem to capture the ideal relationship between two mutually respecting individuals.
Particularly the incorporation of the word “enable” along with “protect” seems to be a hopeful indication that the machine has a human-like wholistic conception of this idea rather than an unbalanced over-protectiveness as feared in some pessimistic scenarios. A neuron like this also may play some role in sycophancy; validating and strengthening the user is pretty precisely sycophancy behavior.
https://openaipublic.blob.core.windows.net/neuron-explainer/neuron-viewer/index.html#/layers/23/neurons/5243
This is part of a project where I am using a low dimensional embedding of GPT2-XL’s 24th layer to assist in interpreting the neurons. The embedding aids in interpretation for more difficult neurons, but this one was pretty straightforward. That said, the embedding does make me suspect this neuron is at least partially capturing romantic relationships because it is near to the erotic story neurons I discussed in a quick-take a couple days ago.
I am trying to gauge interest in a longer frontpage post which will discuss larger findings and methodological details so any encouragement or expression of interest would be appreciated.