Impressive and interesting results; it looks like NLA is still in its early stages, and there is room for improvement from SFT training to evaluation.
Wei Shi
Karma: 1
I got it, thank you very much!
We trained a crosscoder of width 16,384 on the residual stream activations from the middle layer of the Gemma-2 2B base and IT models.
I don’t understand the training process here, as well as the mini-paper from Anthropic. How do you train one crosscoder on the residual stream from two different models?
Really enjoyed this—especially the cross-model transfer result and the “vibe-confabulation” framing. The fact that Qwen and Llama verbalizations remain mutually intelligible makes the textual bottleneck feel much more semantic than I expected.
One thing I’m curious about: how much of the cross-model FVE could be recovered from a similarly styled, context-only summary? That might give a rough decomposition between “summarizing the prompt” and information genuinely specific to the activation.
I also liked the steering-vector experiments. Since a raw difference vector is somewhat off-manifold, I wonder whether comparing AV(h + αv) against AV(h) across several contexts would produce cleaner and more consistent interpretations.
The snippet behavior is fascinating too. My impression is that banning quote characters suppresses the format more than the underlying behavior—the AV still wants to communicate through imagined examples. It would be interesting to see what happens if an NLA is trained from the beginning without snippets.
Thanks for writing it up!