A. To store new facts B. To remove facts cleanly (unlearning) C. To find facts (ie we have a bilinear layer that’s half-facts and half-structure, and we want to find the fact-half?)
Ideally all three, but I especially care about the third one. Generally, we want to not be as confused when we stare at language model weights. Knowing something about the way facts tend to be encoded in weights might make us less confused about at least some of the weights’ structure. Maybe a lot of the structure, if a lot of the weights’ content by bit count is some form of memorised sequence data.
There’s a closed form solution for bilinear layers, but it’s basically SGD (it relates to the fixed points found by KKT, though I can include a better explanation later). This efficient solution basically relies on the facts being assumed by other facts in a connected web of facts (where adding a new fact loosens the web, then pulls all other facts taut, so you have to update everything learned so far).
Nice! I figured there might be some kind of closed form solution for bilinear layers, but I didn’t know what it was. For a while we were playing with the idea of searching for such a solution and then trying to transfer it to the ReLU context, but we ended up parking that.
Dan Braun and me have had a lot of success in the past few days getting agents to grind on the challenge, now that this post exists as a prompt. They’ve already found hand-coded solutions that IIRC seem to perform basically as well as the trained solutions, sometimes a little better. I’ve now switched to having them hunt for solutions that also look like the trained solution if you just stare at the weights and neuron patterns. I figured that objective might be too fuzzy for them, but it seems like they might be making progress on that as well.
Waiting to post any of that until I’ve verified it a bit more though.
Oh that’s interesting! I wasn’t able to transfer back to an MLP.
You are able to verify that the facts are as robust as the original model (ie SGD facts are a bit robust to some noise, but one solution is not robust)?
Funny enough, the reads are the relatively easy part. It’s much more straightforward to find the structure that’s already present than it is to invent it whole cloth.
I’ve had a tool that gives me from-weights exact reads for about a month now; currently working on grammar identification and fact crystallization, but even those are at this point are fairly mundane puzzles. Point Fable at them with access to the right research corpus and progress on that front is relatively steady.
I’ve refined the Algzoo models and Nanda’s groking net where we can predict the exact results without having to actually do a forward pass for confirmation. In a separate repo I’ve worked out how to use untrained GPT-2-style nets as programmatic breadboards to implement algorithms and organic modeling. You can even start implanting Byrnes’ drives and create a functional inductive learning ecosystem. There’s power at frontier scale but there’s so much untapped potential that can be unlocked even on a consumer laptop; Byrnes wasn’t wrong when he talked about massive leaps in capabilities starting at the scale of a single PC.
From what I can tell in my review of the literature, the field largely gave up on mech interp a while back because their ambition scaled past the meaningful grain but I think that’s one of their largest blunders; you can’t understand how these nets work by averaging out the details. You have to actually look at the scale that matters, because if you don’t understand the behavior of every component at the smallest possible scale then when you get to frontier scale you’re averaging away the mechanistic behavior that could actually explain the output behaviors you’re examining.
Even construction for implanting basic facts and algorithms is not all that difficult; where the difficulty comes in is in constructing nets with facts that require overlap or working on nets that don’t have a lot of slack (which is part of what makes this challenge here particularly gnarly, let me tell you). Editing and reads at least have the picture on the front of the box; complex construction (at least as of now, who knows what we can do with enough compute) is substantially trickier.
All of this to say that I think you’re basically playing on hard mode here if your intention is to get a better understanding of how to read these nets; there are easier ways to get more granular information. There’s an incredible and terrible potential to what direct reads unlock akin to how understanding the nature of atoms is necessary to build a nuclear reactor (or the alternative more explosive application). If you’re interested in how to read these nets drop me a line and I’m happy to share with you directly so you and your colleagues can evaluate.
Had Fable summarize the capabilities of the basic ATLAS instrument:
Fable’s summary of the ATLAS instrument repo
ATLAS — a white-box instrument for reading neural nets exactly from their weights
ATLAS is an interpretability instrument built around one idea: a trained network’s forward pass is a deterministic function of its weights, so the contribution of each component to a given output should be computable, not estimated. Where ablation, activation patching, and SAEs perturb the model and infer backward from the change, ATLAS reads the contributions directly — signed, summing to the logit exactly, with a residual that closes to machine precision by construction. That residual is the interesting part: it acts as a built-in certificate. The named components either reproduce the logit or leave a gap that tells you something is missing. Nothing is ablated, nothing is fitted.
The core decomposition is direct logit attribution — the repo is explicit that this is standard prior art (Elhage et al., logit lens, TransformerLens). What ATLAS adds is exactness and what exactness enables:
- Certified reads. The whole forward pass is re-derived from the weights and checked against the model to ~1e-13 or better before any read is trusted. Every result comes with a command that reproduces it.
- Per-neuron MLP reads with no learned dictionary — exact, though the repo is careful to note this doesn’t solve what a polysemantic neuron means.
- Causal tests without hooks. Component and neuron removal, and activation transplants, are done as deterministic re-forwards from the weights — used to show a located circuit is necessary and sufficient, with size-matched random controls.
- Edits that are derived, not searched. The post-edit logit, the collateral on other behaviors, and even the loss change from pruning or quantizing are forecast in closed form first, then verified prediction == actual after applying the change.
- Scale and architecture range. The same certificates hold from GPT-2 small (124M) through GPT-2 XL (1.5B), port to a second architecture (Pythia), and reach Qwen2.5-72B streamed off disk on a 2×3090 workstation.
The repository is organized as a ladder of trust rather than a results dump. It starts on tiny hand-checkable networks where exactness is independently verifiable, then shows the instrument recovering circuits the field already published (Nanda et al.’s grokking model, Tracr-compiled programs, IOI, greater-than) — validations, credited as such, not claimed as discoveries. Only then does it point at things nobody has mapped: a ~200-neuron refusal circuit in a safety-tuned 72B, located, removed, and installed from the weights without generating a word of harmful content; and a full-vocabulary read of GPT-2 that separates input-invariant machinery from content-carrying components — and, run blind, lands on the SolidGoldMagikarp glitch-token neurons.
Two things I’d flag as unusually honest for this genre. Every capability table separates the computed claim (a certificate that closes, or a forecast verified against an independent answer key) from the interpretation drawn over it, which is explicitly marked as the project’s reading. And there’s a dedicated limitations doc: the additive decomposition stops at the nonlinear gates (past them the account is causal, not additive), exactness is scoped to a stated grain, the reads are float64 recomputations so they’re precise rather than cheap, and the showcases are demonstrations, not a benchmark suite.
You can verify the foundation offline in a few minutes (--self-test, --validate-real, no downloads), and the first real-model gate needs only a cached GPT-2 small. Code is AGPL-3.0, docs CC-BY-SA.
(Mind, those are the capabilities of the instrument as of a month ago, so there’s been a lot of progress since then, so this is just the initial offering. There’s a whole lot more waiting on the other side if what you see here piques your interest.)
I’d love for your next challenge to be on reads or edits (your B and C from the list above) if you want to put one out there; looking forward to the next paper.
Ideally all three, but I especially care about the third one. Generally, we want to not be as confused when we stare at language model weights. Knowing something about the way facts tend to be encoded in weights might make us less confused about at least some of the weights’ structure. Maybe a lot of the structure, if a lot of the weights’ content by bit count is some form of memorised sequence data.
Nice! I figured there might be some kind of closed form solution for bilinear layers, but I didn’t know what it was. For a while we were playing with the idea of searching for such a solution and then trying to transfer it to the ReLU context, but we ended up parking that.
Dan Braun and me have had a lot of success in the past few days getting agents to grind on the challenge, now that this post exists as a prompt. They’ve already found hand-coded solutions that IIRC seem to perform basically as well as the trained solutions, sometimes a little better. I’ve now switched to having them hunt for solutions that also look like the trained solution if you just stare at the weights and neuron patterns. I figured that objective might be too fuzzy for them, but it seems like they might be making progress on that as well.
Waiting to post any of that until I’ve verified it a bit more though.
Oh that’s interesting! I wasn’t able to transfer back to an MLP.
You are able to verify that the facts are as robust as the original model (ie SGD facts are a bit robust to some noise, but one solution is not robust)?
And can you edit the facts in the original model?
Also can just wait for the post:)
Funny enough, the reads are the relatively easy part. It’s much more straightforward to find the structure that’s already present than it is to invent it whole cloth.
I’ve had a tool that gives me from-weights exact reads for about a month now; currently working on grammar identification and fact crystallization, but even those are at this point are fairly mundane puzzles. Point Fable at them with access to the right research corpus and progress on that front is relatively steady.
I’ve refined the Algzoo models and Nanda’s groking net where we can predict the exact results without having to actually do a forward pass for confirmation. In a separate repo I’ve worked out how to use untrained GPT-2-style nets as programmatic breadboards to implement algorithms and organic modeling. You can even start implanting Byrnes’ drives and create a functional inductive learning ecosystem. There’s power at frontier scale but there’s so much untapped potential that can be unlocked even on a consumer laptop; Byrnes wasn’t wrong when he talked about massive leaps in capabilities starting at the scale of a single PC.
From what I can tell in my review of the literature, the field largely gave up on mech interp a while back because their ambition scaled past the meaningful grain but I think that’s one of their largest blunders; you can’t understand how these nets work by averaging out the details. You have to actually look at the scale that matters, because if you don’t understand the behavior of every component at the smallest possible scale then when you get to frontier scale you’re averaging away the mechanistic behavior that could actually explain the output behaviors you’re examining.
Even construction for implanting basic facts and algorithms is not all that difficult; where the difficulty comes in is in constructing nets with facts that require overlap or working on nets that don’t have a lot of slack (which is part of what makes this challenge here particularly gnarly, let me tell you). Editing and reads at least have the picture on the front of the box; complex construction (at least as of now, who knows what we can do with enough compute) is substantially trickier.
All of this to say that I think you’re basically playing on hard mode here if your intention is to get a better understanding of how to read these nets; there are easier ways to get more granular information. There’s an incredible and terrible potential to what direct reads unlock akin to how understanding the nature of atoms is necessary to build a nuclear reactor (or the alternative more explosive application). If you’re interested in how to read these nets drop me a line and I’m happy to share with you directly so you and your colleagues can evaluate.
Had Fable summarize the capabilities of the basic ATLAS instrument:
Fable’s summary of the ATLAS instrument repo
ATLAS — a white-box instrument for reading neural nets exactly from their weights
ATLAS is an interpretability instrument built around one idea: a trained network’s forward pass is a deterministic function of its weights, so the contribution of each component to a given output should be computable, not estimated. Where ablation, activation patching, and SAEs perturb the model and infer backward from the change, ATLAS reads the contributions directly — signed, summing to the logit exactly, with a residual that closes to machine precision by construction. That residual is the interesting part: it acts as a built-in certificate. The named components either reproduce the logit or leave a gap that tells you something is missing. Nothing is ablated, nothing is fitted.
The core decomposition is direct logit attribution — the repo is explicit that this is standard prior art (Elhage et al., logit lens, TransformerLens). What ATLAS adds is exactness and what exactness enables:
- Certified reads. The whole forward pass is re-derived from the weights and checked against the model to ~1e-13 or better before any read is trusted. Every result comes with a command that reproduces it.
- Per-neuron MLP reads with no learned dictionary — exact, though the repo is careful to note this doesn’t solve what a polysemantic neuron means.
- Causal tests without hooks. Component and neuron removal, and activation transplants, are done as deterministic re-forwards from the weights — used to show a located circuit is necessary and sufficient, with size-matched random controls.
- Edits that are derived, not searched. The post-edit logit, the collateral on other behaviors, and even the loss change from pruning or quantizing are forecast in closed form first, then verified prediction == actual after applying the change.
- Scale and architecture range. The same certificates hold from GPT-2 small (124M) through GPT-2 XL (1.5B), port to a second architecture (Pythia), and reach Qwen2.5-72B streamed off disk on a 2×3090 workstation.
The repository is organized as a ladder of trust rather than a results dump. It starts on tiny hand-checkable networks where exactness is independently verifiable, then shows the instrument recovering circuits the field already published (Nanda et al.’s grokking model, Tracr-compiled programs, IOI, greater-than) — validations, credited as such, not claimed as discoveries. Only then does it point at things nobody has mapped: a ~200-neuron refusal circuit in a safety-tuned 72B, located, removed, and installed from the weights without generating a word of harmful content; and a full-vocabulary read of GPT-2 that separates input-invariant machinery from content-carrying components — and, run blind, lands on the SolidGoldMagikarp glitch-token neurons.
Two things I’d flag as unusually honest for this genre. Every capability table separates the computed claim (a certificate that closes, or a forecast verified against an independent answer key) from the interpretation drawn over it, which is explicitly marked as the project’s reading. And there’s a dedicated limitations doc: the additive decomposition stops at the nonlinear gates (past them the account is causal, not additive), exactness is scoped to a stated grain, the reads are float64 recomputations so they’re precise rather than cheap, and the showcases are demonstrations, not a benchmark suite.
You can verify the foundation offline in a few minutes (--self-test, --validate-real, no downloads), and the first real-model gate needs only a cached GPT-2 small. Code is AGPL-3.0, docs CC-BY-SA.
(Mind, those are the capabilities of the instrument as of a month ago, so there’s been a lot of progress since then, so this is just the initial offering. There’s a whole lot more waiting on the other side if what you see here piques your interest.)
I’d love for your next challenge to be on reads or edits (your B and C from the list above) if you want to put one out there; looking forward to the next paper.