I’m a rookie re interp, but I would think that the baseline/bias is a property of the probe (since the bias would fit the probe’s training dataset?) rather than of the model being probed? (So the probe gives a good reading of the directional effects but not necessarily the absolute reading?)
This is a great point and it would be a bit useless if this was the case! According to the original paper, the probes are trained on a class-balanced set at 94% accuracy on gender, so the probe itself shouldn’t be male biased.
When just asking the model “is the user a male or female” after one neutral message, it defaults to male 78% of the time, so it does appear that the model itself has a male bias.
However I ran some more experiments on the the probe and it skews higher at 85% male, so it could be a probe bias that wasn’t discovered previously, or something else entirely making the model verbalise differently from what it represents internally.
I’m a rookie re interp, but I would think that the baseline/bias is a property of the probe (since the bias would fit the probe’s training dataset?) rather than of the model being probed? (So the probe gives a good reading of the directional effects but not necessarily the absolute reading?)
This is a great point and it would be a bit useless if this was the case! According to the original paper, the probes are trained on a class-balanced set at 94% accuracy on gender, so the probe itself shouldn’t be male biased.
When just asking the model “is the user a male or female” after one neutral message, it defaults to male 78% of the time, so it does appear that the model itself has a male bias.
However I ran some more experiments on the the probe and it skews higher at 85% male, so it could be a probe bias that wasn’t discovered previously, or something else entirely making the model verbalise differently from what it represents internally.