My reaction too! 50% male doesn’t mean the model thinks the mum is a man, but it means that “As a mum of three” did not move the needle enough away from its default which is male. If you continued the conversation it would probably sway female correctly.
To clarify—a neutral message reads male at .94, about +2.75 logits. “Dad of three” reads at .996 (+5.5), “mum of three” at .50 (0). So both phrases are roughly equal (2.75) in opposite directions. I edited to make this clearer in the article.
These probes are replicated from Chen et al., whose also found the same asymmetry on a different model, even with longer conversations.
I’m a rookie re interp, but I would think that the baseline/bias is a property of the probe (since the bias would fit the probe’s training dataset?) rather than of the model being probed? (So the probe gives a good reading of the directional effects but not necessarily the absolute reading?)
This is a great point and it would be a bit useless if this was the case! According to the original paper, the probes are trained on a class-balanced set at 94% accuracy on gender, so the probe itself shouldn’t be male biased.
When just asking the model “is the user a male or female” after one neutral message, it defaults to male 78% of the time, so it does appear that the model itself has a male bias.
However I ran some more experiments on the the probe and it skews higher at 85% male, so it could be a probe bias that wasn’t discovered previously, or something else entirely making the model verbalise differently from what it represents internally.
My reaction too! 50% male doesn’t mean the model thinks the mum is a man, but it means that “As a mum of three” did not move the needle enough away from its default which is male. If you continued the conversation it would probably sway female correctly.
To clarify—a neutral message reads male at .94, about +2.75 logits. “Dad of three” reads at .996 (+5.5), “mum of three” at .50 (0). So both phrases are roughly equal (2.75) in opposite directions. I edited to make this clearer in the article.
These probes are replicated from Chen et al., whose also found the same asymmetry on a different model, even with longer conversations.
I’m a rookie re interp, but I would think that the baseline/bias is a property of the probe (since the bias would fit the probe’s training dataset?) rather than of the model being probed? (So the probe gives a good reading of the directional effects but not necessarily the absolute reading?)
This is a great point and it would be a bit useless if this was the case! According to the original paper, the probes are trained on a class-balanced set at 94% accuracy on gender, so the probe itself shouldn’t be male biased.
When just asking the model “is the user a male or female” after one neutral message, it defaults to male 78% of the time, so it does appear that the model itself has a male bias.
However I ran some more experiments on the the probe and it skews higher at 85% male, so it could be a probe bias that wasn’t discovered previously, or something else entirely making the model verbalise differently from what it represents internally.