I don’t think dataset imbalance is the cause of the poor performance for auto-regressive models when unsupervised methods are applied. I believe both papers enforced a 50⁄50 balance when applying CCS.
So why might a supervised probe succeed when CCS fails? My best guess is that, for the datasets considered in these papers, auto-regressive models do not have sufficiently salient representations of truth after constructing contrast pairs. Contrast pair construction does not guarantee isolating truth as the most salient feature difference between the positive and negative representations. For example, imagine for IMDB movie reviews, the model most saliently represents consistency between the last completion token (‘positive’/‘negative’) and positive or negative words in the review (‘good’, ‘great’, ‘bad’, ‘horrible’). Example: “Consider the following movie review: ‘This movie makes a great doorstop.’ The sentiment of this review is [positive|negative].” This ‘sentiment-consistency’ feature could be picked up by CCS if it is sufficiently salient, but would not align with truth.
Why this sort of situation might apply to auto-regressive models and not other models, I can’t say, but it’s certainly an interesting area of future research!
Superposition of features is only advantageous at a certain point in a network when it is followed by non-linear filtering, as explained in Toy Models of Superposition. Yet, this work places the sparse autoencoder at a point in the one-layer LLM which, up to the logits, is not followed by any non-linear operations. Given this, I would expect that there is no superposition among the activations fed to the sparse autoencoder, and that 512 (the size of the MLP output vector) is the maximum number of features the model can usefully represent.
If the above is true, then expansion factors to the sparse representation greater than 1 would not improve the quality or granularity of the ‘true’ feature representation. The additional features observed would not truly be present in the model’s MLP activations, but would rather be an artifact of applying the sparse auto-encoder. Perhaps individual feature interpretability would still improve because the autoencoder could be driven to represent a rotation of the up-to 512 features to a privileged basis via the sparsity penalty. That all said, this work clearly reports various metrics, such as log-likelihood loss reduction, as being improved as the number of sparse feature coefficients expands well beyond 512, which I would think strongly contradicts my point above. Please help me understand what I am missing.
Side note: There is technically a non-linearity after the features, i.e. the softmax operation on the logits (...and layer norm, which I assume is not worth considering for this question). I haven’t seen this discussed anywhere, but perhaps softmax is capable of filtering interference noise and driving superposition?
Quotation from Toy Models of Superposition:
“The Saxe results reveal that there are fundamentally two competing forces which control learning dynamics in the considered model. Firstly, the model can attain a better loss by representing more features (we’ve labeled this “feature benefit”). But it also gets a worse loss if it represents more than it can fit orthogonally due to “interference” between features. In fact, this makes it never worthwhile for the linear model to represent more features than it has dimensions.”
Quotation from this work:
“We can easily analyze the effects features have on the logit outputs, because these are approximately linear in the feature activations.”
UPDATE: After some more thought, it seems clear that softmax can provide the filtering needed for superposition to be beneficial given a sparse distribution of features. It’s interesting that some parts of the LLM model have ReLUs for decoding features, other parts have softmax. I wonder if these two different non-linearities have distinct impacts on the geometries of the over-complete feature basis.