Hi everyone! I am a Junior Computer Science student in Pennsylvania. I’ve been interested in mechanistic interpretability since Freshman year, but to be honest, most of my actual college experience has been in other areas (I’ve done some physics research and light ML development) and I am only really getting serious about interp now, after circling it for about two years. So I am very much at the beginning here, and looking for some guidance.
To make that concrete: this summer I’ve made my way through Karpathy’s Zero-to-Hero course and read through Elhage et al.’s “A Mathematical Framework for Transformer Circuits.” To force myself to actually build something, I trained a small char-level transformer on Nietzsche and tried to reverse-engineer it straight from the weights. I think I found a genuine induction head in the last layer; the copying score (roughly, how strongly a head reproduces the token it attends to) spikes on one head, and the QK side held up when I ran a content-swap check. However, I did all of this in raw PyTorch, so I am sure I am missing a lot of the standard practice.
Which is really why I am here. I don’t have a good map of where the field actually is right now. I’ve heard TransformerLens is what everyone uses (I haven’t touched it yet), and I’ve heard SAEs / feature circuits might be where a lot of the current work is. But I honestly don’t know if it’s the center of the field, one of several branches, or already old news. So for someone at my stage: is the field concentrated around a few directions I should orient toward, or is it branched enough that I should just pick one and go? And does “Indirect Object Identification (IOI) in TransformerLens, then dip into SAEs” sound like a reasonable on-ramp, or is there a better one?
But overall, I am excited to be part of this community, and to push this beyond a hobby.
Hi everyone! I am a Junior Computer Science student in Pennsylvania. I’ve been interested in mechanistic interpretability since Freshman year, but to be honest, most of my actual college experience has been in other areas (I’ve done some physics research and light ML development) and I am only really getting serious about interp now, after circling it for about two years. So I am very much at the beginning here, and looking for some guidance.
To make that concrete: this summer I’ve made my way through Karpathy’s Zero-to-Hero course and read through Elhage et al.’s “A Mathematical Framework for Transformer Circuits.” To force myself to actually build something, I trained a small char-level transformer on Nietzsche and tried to reverse-engineer it straight from the weights. I think I found a genuine induction head in the last layer; the copying score (roughly, how strongly a head reproduces the token it attends to) spikes on one head, and the QK side held up when I ran a content-swap check. However, I did all of this in raw PyTorch, so I am sure I am missing a lot of the standard practice.
Which is really why I am here. I don’t have a good map of where the field actually is right now. I’ve heard TransformerLens is what everyone uses (I haven’t touched it yet), and I’ve heard SAEs / feature circuits might be where a lot of the current work is. But I honestly don’t know if it’s the center of the field, one of several branches, or already old news. So for someone at my stage: is the field concentrated around a few directions I should orient toward, or is it branched enough that I should just pick one and go? And does “Indirect Object Identification (IOI) in TransformerLens, then dip into SAEs” sound like a reasonable on-ramp, or is there a better one?
But overall, I am excited to be part of this community, and to push this beyond a hobby.