Really enjoyed this—especially the cross-model transfer result and the “vibe-confabulation” framing. The fact that Qwen and Llama verbalizations remain mutually intelligible makes the textual bottleneck feel much more semantic than I expected.
One thing I’m curious about: how much of the cross-model FVE could be recovered from a similarly styled, context-only summary? That might give a rough decomposition between “summarizing the prompt” and information genuinely specific to the activation.
I also liked the steering-vector experiments. Since a raw difference vector is somewhat off-manifold, I wonder whether comparing AV(h + αv) against AV(h) across several contexts would produce cleaner and more consistent interpretations.
The snippet behavior is fascinating too. My impression is that banning quote characters suppresses the format more than the underlying behavior—the AV still wants to communicate through imagined examples. It would be interesting to see what happens if an NLA is trained from the beginning without snippets.
Really enjoyed this—especially the cross-model transfer result and the “vibe-confabulation” framing. The fact that Qwen and Llama verbalizations remain mutually intelligible makes the textual bottleneck feel much more semantic than I expected.
One thing I’m curious about: how much of the cross-model FVE could be recovered from a similarly styled, context-only summary? That might give a rough decomposition between “summarizing the prompt” and information genuinely specific to the activation.
I also liked the steering-vector experiments. Since a raw difference vector is somewhat off-manifold, I wonder whether comparing AV(h + αv) against AV(h) across several contexts would produce cleaner and more consistent interpretations.
The snippet behavior is fascinating too. My impression is that banning quote characters suppresses the format more than the underlying behavior—the AV still wants to communicate through imagined examples. It would be interesting to see what happens if an NLA is trained from the beginning without snippets.
Thanks for writing it up!