Reminds me of [eigenwords](https://www.cis.upenn.edu/~ungar/eigenwords/) from way back when research was on word embeddings, which in makes me consider how word embedding work (Miklov etc) was like a useful precursor for interpretability in only mildly related seq2seq models (RNNs, GPTs, etc).
We’re in a similar stage of research with interpretability—many mechinterp are variations on “take a word sequence, see what activates, do it at scale, cluster the activations;” this feels somewhat analogous to BoW/Skipgram approaches for word embeddings.
seq2seq models ended up prevailing over word embedding models as they learned latent representations of words implicitly (or you might say that seq2seq was a generalization of word embeddings), which begs the question: what will the generalizations be over current activation-based mech-interp, eg probes? I think [Anthropic replacement models](https://transformer-circuits.pub/2025/attribution-graphs/methods.html) is a move that way.
More explicitly, the structural mapping is: (learn word embeddings explicitly) : (learn word embeddings latently from seq2seq) :: (learn activations explicitly) : (learn activations latently from replacement models).
Reminds me of [eigenwords](https://www.cis.upenn.edu/~ungar/eigenwords/) from way back when research was on word embeddings, which in makes me consider how word embedding work (Miklov etc) was like a useful precursor for interpretability in only mildly related seq2seq models (RNNs, GPTs, etc).
We’re in a similar stage of research with interpretability—many mechinterp are variations on “take a word sequence, see what activates, do it at scale, cluster the activations;” this feels somewhat analogous to BoW/Skipgram approaches for word embeddings.
seq2seq models ended up prevailing over word embedding models as they learned latent representations of words implicitly (or you might say that seq2seq was a generalization of word embeddings), which begs the question: what will the generalizations be over current activation-based mech-interp, eg probes? I think [Anthropic replacement models](https://transformer-circuits.pub/2025/attribution-graphs/methods.html) is a move that way.
More explicitly, the structural mapping is: (learn word embeddings explicitly) : (learn word embeddings latently from seq2seq) :: (learn activations explicitly) : (learn activations latently from replacement models).