Intuition pump: When you double the size of the residual stream, you get a squaring in the number of distinct (almost-orthogonal) features that a language model can learn. If we call a model X and its twice-as-wide version Y, we might expect that the features in Y all look like Cartesian pairs of features in X.
Re-stating an important point: this suggests that feature splitting could be something inherent to language models as opposed to an artefact of SAE training.
I suspect this could be elucidated pretty cleanly in a toy model. Need to think more about the specific toy setting that will be most appropriate. Potentially just re-using the setup from Toy Models of Superposition is already good enough.
Related idea: If we can show that models themselves learn featyres of different granularity, we could then test whether SAEs reflect this difference. (I expect they do not.) This would imply that SAEs capture properties of the data rather than the model.
Intuition pump: When you double the size of the residual stream, you get a squaring in the number of distinct (almost-orthogonal) features that a language model can learn. If we call a model X and its twice-as-wide version Y, we might expect that the features in Y all look like Cartesian pairs of features in X.
Re-stating an important point: this suggests that feature splitting could be something inherent to language models as opposed to an artefact of SAE training.
I suspect this could be elucidated pretty cleanly in a toy model. Need to think more about the specific toy setting that will be most appropriate. Potentially just re-using the setup from Toy Models of Superposition is already good enough.
Related idea: If we can show that models themselves learn featyres of different granularity, we could then test whether SAEs reflect this difference. (I expect they do not.) This would imply that SAEs capture properties of the data rather than the model.