The paper argues that there is one generalizing truth direction which corresponds to whether a statement is true, and one polarity-sensitive truth direction that corresponds to , related to Sam Marks’ work on LLMs representing XOR-features. It further states that the truth directions for affirmative and negated statements are linear combinations of and , just with different coefficients.
Is there evidence that is an actual, elementary feature used by the language model, and not a linear combination of other features? For example, I could imagine that is a linear combination of features like e.g., or , … .
Do you think we have reason to believe that is an elementary feature, and not a linear combination?
If the latter is the case, it seems to me that there is high risk of the probe failing when the distribution changes (e.g. on french text in the example above), particularly with XOR-features that change polarity.
Unfortunately, GLM-5 was too large for the setup I used when I ran the experiments. I won’t run it, but I’d also be interested in case somebody else does try.