Yeah, this makes a ton of sense. Thx for taking the time to give it a closer look and also your detailed response :)
So then in order for the SAE to be useful I’d have to train it on a lot of sentiment data and then I could maybe discover some interpretable sentiment related features that could help me understand why a model thinks a review is positive/negative...
I’m not quite sure what you mean with “the sentiment will not be linearly separable”.
The hidden states are linearly separable (to some extend), but the sparse representations perform worse than the original representations in my experiment.
I am training logistic regression classifiers on the original, and sparse representations respectively, so I am multiplying the residual stream states (and their sparse encodings) with weights. These weights could (but don’t have to) align with some meaningful direction like hidden_states(“positive”)-hidden_states(“negative”).
I’m not sure if I understood your comment about the logit lens. Are you proposing this as an alternative way of testing for linear separability? But then shouldn’t the information already be encoded in the hidden states and thus extractable with a classifier?
Thx for the feedback. Fixed typo and added ITI reference.
The relative difference in the train accuracies looks pretty similar. But yeah, @SenR already pointed to the low number of active features in the SAE, so that explains this nicely.