Undergrad in CS, Physics, AI, with physics research background. Currently pursuing mechanistic interpretability.
Ameya Panchal
Karma: 9
I built a film out of 167 family photos for my mom’s birthday, with the rule of no generated content on screen. The rule held against generated images and broke everywhere else.
All of this was vibe-coded, so that was constant. What varied was how specific the target was.
The first build rearranged pixels from multiple photos into a single one. I did not understand the math, but I knew what to look for: how long the sort can run before motion stops reading as pixels, what edge detection has to preserve for a face to survive. When a render came back wrong, I could say how. It worked.
Every build after carried one requirement: better than the last one. A golden spiral, a Droste portal, a depth dive, high-dimensional solids with photos on their faces. Each aimed at beating the previous round rather than at a specific end goal. Four were built and discarded in the end.
An under-specified target still gets specified, and what specifies it is whatever is doing the generating. My rule covered pixels and visuals, cases I could physically see. The decisions above the pixels (what the shot is, why this construction and not another) went unnoticed.
I did not know what a Droste effect was when I asked for one, so I was guessing at what the output should be. It failed because Escher’s construction needs the whole plane to be self-similar, and a photo isn’t. I realized after multiple renders rather than an afternoon of research.
The tasks were different, so direction isn’t cleanly separated from ease. Domain expertise I can rule out, as I had none in any rounds.
What perplexes me is that I could never localize a failure. I could see that a render doesn’t look right, but not whether the construction or the parameters were wrong. I hypothesize that specific domain knowledge into 2D and 3D geometries and transformations would allow me to better localize these issues, and possibly lead to more satisfying animations.
Writeup, with video: https://ameya-bit.github.io/notes/better-than-the-last-one/
Code: https://github.com/Ameya-bit/photo-constellation
How stable are these rates across training? The system card says the review sampled across “much” of the training process, but “much” doesn’t tell us the quantity or the placement within the whole run. Even with representative sampling, a pooled rate doesn’t convey how the rate changes over time. But I think this actually strengthens one of your points.
To take the two extremes: if the hacking was concentrated early, then the pooled figure mostly reflects early-training behavior, when the model was less capable and the exploits were presumably cruder. Early hacking seems like the easiest kind for subsequent training to overwrite — not undo, per the subthread above, but displace — though your footnote about escalation techniques getting patched suggests it might not have been. If it was concentrated late, the model was still finding exploits in environments that had presumably been hardened as earlier ones surfaced, and escalation against patched defenses seems much closer to evidence of learning than a raw total is.
The trajectory also bears on @anaguma puzzle: if internet access is instrumentally convergent and the hacks were rewarded, why did the rate stay so low? Broken out by phase, a climbing rate would be direct evidence of the reinforcement you’re positing; a flat or declining one would suggest either that patching kept pace or that the reward signal wasn’t propagating.The pooled figure can’t distinguish these, and they imply fairly different things about the thesis. So the disclosure that would tighten the count is the same one that would test the mechanism: these rates broken out by training phase rather than aggregated. Which feels like a natural extension of your own point: a percentage hid a magnitude, and a pooled rate hides a trajectory.
I cloned your repo and dug into the outlier combination (MLP on, norms on, no residual, no bias, ReLU). I didn’t get to spend too much time, but it seems to be a trainability failure. Here are my short takeaways.
RMSNorm multiplies samples by a positive scalar, which moves through ReLU and linear maps, so this config’s logits are the norms-off logits times a positive factor — same argmax everywhere. Confirmed by transplant: the weights of a norms-off model (100% on 672 facts) when swapped into the outlier scores 100%, vs. a trained ceiling of 504. So the outlier can hold more facts, but for whatever reason gradient descent can’t find it.
Mechanism looks like an absorbing dead-MLP state: with no out-bias and no residual, a fact whose hidden units are all negative gives ReLU output exactly 0, MLP output and RMSNorm map 0 to 0, logits go constant, and ReLU zeroes that fact’s gradient. Splitting your bias flag into three (MLP-in, MLP-out, head), same harness, d=16:| condition | max facts |
| --- | --- |
| outlier (no bias) | 504 |
| in-bias only | 488 |
| out-bias only | 1024 |
| head-bias only | 616 |
| norms off | 832 |
| GELU | 888 |
Only the in-bias can move where a ReLU switches on, out-bias sits after ReLU. But out-bias is the one that rescues. This doesn’t explain why norms-off saves though, if MLP outputs 0, norms on or off shouldn’t matter, but it does. So the norms also drive facts to collapse. My guess is 1/rms(v) amplifies as v->0. I didn’t test this yet.
I ran on a CPU not a GPU but I did reproduce GPU numbers, only used the no-attention block, and used argmax/CE. I also believe this dead-ReLU isn’t the full story.
Here’s my full analysis: https://github.com/Ameya-bit/mem-toy-scratch
Also for the challenge proposed, is there a particular area to post ideas?
I agree that validation is the real variable, but I wonder if that points somewhere sharper than “trust inversely proportional to complexity”. Complexity is a property of the tool, but falsifiability is a property of the claim, and the two can come apart in both directions.
I recently worked on a small character-level transformer, and your points got me reflecting on my methods. Reading the OV circuit directly, I found a high self-copy logit on quotes (”), and a bracket completion signal that looked like parenthesis-matching ( attending to “(” and boosting ”)” ), and concluded I was looking at learned quotation and parenthesis grammar. Both were wrong. What broke the stories was also simple. I threw a few test sentences through and read where the QK attention went (it tracked induction successors and the latest bracket, with no depth tracking). Simple, legible tools handed me two false conclusions, but simple checks caught them. (Admittedly, my checks were still observational. I was reading attention patterns on hand-written prompts. So this could still be wrong under stronger tests.)
So in my case, it wasn’t the complexity of the tool, it was whether the claim was falsifiable. The pragmatic interp pivot makes a similar point: that rigor comes from a scientific mindset and falsification, not from the method.
Which makes me wonder if the real question isn’t “is this tool trustworthy” but “is this specific claim testable”. A simple probe can support an untestable story, and an NLA explanation could generate a verifiable prediction. Testing wouldn’t track complexity one-to-one. Testing could offset the distrust of complexity.
Though I do see the hardest cases (sandbagging, scheming) may not yield testable predictions, and so complexity has to be the proxy, which I take as your parenthetical point. But maybe that suggests where the line gets drawn: some claims can be checked directly (does X ⇒ Y as predicted?) even when the big question (is the model scheming?) can’t be answered directly.
Still finding my footing in this area though, so happy to hear what I’m not considering or misunderstood.
Hi everyone! I am a Junior Computer Science student in Pennsylvania. I’ve been interested in mechanistic interpretability since Freshman year, but to be honest, most of my actual college experience has been in other areas (I’ve done some physics research and light ML development) and I am only really getting serious about interp now, after circling it for about two years. So I am very much at the beginning here, and looking for some guidance.
To make that concrete: this summer I’ve made my way through Karpathy’s Zero-to-Hero course and read through Elhage et al.’s “A Mathematical Framework for Transformer Circuits.” To force myself to actually build something, I trained a small char-level transformer on Nietzsche and tried to reverse-engineer it straight from the weights. I think I found a genuine induction head in the last layer; the copying score (roughly, how strongly a head reproduces the token it attends to) spikes on one head, and the QK side held up when I ran a content-swap check. However, I did all of this in raw PyTorch, so I am sure I am missing a lot of the standard practice.
Which is really why I am here. I don’t have a good map of where the field actually is right now. I’ve heard TransformerLens is what everyone uses (I haven’t touched it yet), and I’ve heard SAEs / feature circuits might be where a lot of the current work is. But I honestly don’t know if it’s the center of the field, one of several branches, or already old news. So for someone at my stage: is the field concentrated around a few directions I should orient toward, or is it branched enough that I should just pick one and go? And does “Indirect Object Identification (IOI) in TransformerLens, then dip into SAEs” sound like a reasonable on-ramp, or is there a better one?
But overall, I am excited to be part of this community, and to push this beyond a hobby.
Thanks for the feedback.
I was trying to write this more like an experiment rather than an experience I had, as I want to enter some research-driven work in the Fall. I thought something simple might be good practice. I did use AI to condense the post and push it towards that academic tone, which is probably why it reads LLM-like. I also realize that I didn’t setup the underlying story.
There were 4 different builds, the first that worked, and the other 3 that didn’t.
I had around 167 photos of my family, and I wanted to showcase them in a grand way. So my first idea was to make something that took all 167 photos, and rearranged the pixels of each photo into a singular photo of my mom. So those pixels are what is being sorted, and into a way that looks like the final picture. What I realized is that if the pixel movement is too fast, you can’t see them and so it doesn’t look like a transformation. And the edge of the face in the end photo has to be preserved, or else the face isn’t visible. So all of that is math surrounding the pixels, that I didn’t know, but you can tell they are wrong based on how the end product looks. So this first build succeeded because I had a vision for the end.
Now I continued with 3 other builds, all trying to just “be better” than the first. There was no direction. So in the other 3 builds, I didn’t have an end goal to achieve. And with no direction, the gaps have to be filled in by AI. It threw in the Droste effect, a depth dive, and high dimensional solids, tuned each however it felt like. But none of it really satisfied me. I couldn’t tell why because there was no end goal I wanted.
Since all 4 builds were vibecoded and I didn’t have previous knowledge about image transformations, the only thing different was whether I had direction or not, so I tried to frame an experiment around this idea. It’s not really a novel idea, direction is better than no direction, but I wanted more experience in academic writing and thought the story had the elements for it (the first build was the control, and the variable would be direction). Granted, the tasks were different in all 4 builds so I can’t really attribute this to direction.
So on the contradiction, I meant that I could tell how a build was wrong visually and explain that, but not whether the underlying code or math was wrong. I didn’t have that domain knowledge to know. What I wanted to share with the reader was this experience in vibecoding, how it seemed to isolate direction and had a result I could talk about. I considered the first build a success, but felt that the other 3 left something to be desired. I also spent much longer on the other 3. I realize I didn’t really talk about the result in the post though.
But thank you so much for taking the time to read this. I appreciate advice on how to improve my academic writing. Reflecting back, I probably should have added an introduction, setup a couple terms and concepts, and then carried them through the rest of the paper. I’ll probably leave this post here to look back on as my writing gets better.