Undergrad in CS, Physics, AI, with physics research background. Currently pursuing mechanistic interpretability.
Ameya Panchal
Karma: 1
An Induction Head in Disguise: Chasing Grammar in a Character-Level Transformer
Hi everyone! I am a Junior Computer Science student in Pennsylvania. I’ve been interested in mechanistic interpretability since Freshman year, but to be honest, most of my actual college experience has been in other areas (I’ve done some physics research and light ML development) and I am only really getting serious about interp now, after circling it for about two years. So I am very much at the beginning here, and looking for some guidance.
To make that concrete: this summer I’ve made my way through Karpathy’s Zero-to-Hero course and read through Elhage et al.’s “A Mathematical Framework for Transformer Circuits.” To force myself to actually build something, I trained a small char-level transformer on Nietzsche and tried to reverse-engineer it straight from the weights. I think I found a genuine induction head in the last layer; the copying score (roughly, how strongly a head reproduces the token it attends to) spikes on one head, and the QK side held up when I ran a content-swap check. However, I did all of this in raw PyTorch, so I am sure I am missing a lot of the standard practice.
Which is really why I am here. I don’t have a good map of where the field actually is right now. I’ve heard TransformerLens is what everyone uses (I haven’t touched it yet), and I’ve heard SAEs / feature circuits might be where a lot of the current work is. But I honestly don’t know if it’s the center of the field, one of several branches, or already old news. So for someone at my stage: is the field concentrated around a few directions I should orient toward, or is it branched enough that I should just pick one and go? And does “Indirect Object Identification (IOI) in TransformerLens, then dip into SAEs” sound like a reasonable on-ramp, or is there a better one?
But overall, I am excited to be part of this community, and to push this beyond a hobby.
I agree that validation is the real variable, but I wonder if that points somewhere sharper than “trust inversely proportional to complexity”. Complexity is a property of the tool, but falsifiability is a property of the claim, and the two can come apart in both directions.
I recently worked on a small character-level transformer, and your points got me reflecting on my methods. Reading the OV circuit directly, I found a high self-copy logit on quotes (”), and a bracket completion signal that looked like parenthesis-matching ( attending to “(” and boosting ”)” ), and concluded I was looking at learned quotation and parenthesis grammar. Both were wrong. What broke the stories was also simple. I threw a few test sentences through and read where the QK attention went (it tracked induction successors and the latest bracket, with no depth tracking). Simple, legible tools handed me two false conclusions, but simple checks caught them. (Admittedly, my checks were still observational. I was reading attention patterns on hand-written prompts. So this could still be wrong under stronger tests.)
So in my case, it wasn’t the complexity of the tool, it was whether the claim was falsifiable. The pragmatic interp pivot makes a similar point: that rigor comes from a scientific mindset and falsification, not from the method.
Which makes me wonder if the real question isn’t “is this tool trustworthy” but “is this specific claim testable”. A simple probe can support an untestable story, and an NLA explanation could generate a verifiable prediction. Testing wouldn’t track complexity one-to-one. Testing could offset the distrust of complexity.
Though I do see the hardest cases (sandbagging, scheming) may not yield testable predictions, and so complexity has to be the proxy, which I take as your parenthetical point. But maybe that suggests where the line gets drawn: some claims can be checked directly (does X ⇒ Y as predicted?) even when the big question (is the model scheming?) can’t be answered directly.
Still finding my footing in this area though, so happy to hear what I’m not considering or misunderstood.