Returning to ARC
I’ve returned to the Alignment Research Center (ARC) as executive director. My main focus for the next six months will be driving forward ARC’s research agenda—building techniques to find mechanistic explanations for neural network behavior and then using those explanations to detect and address misalignment. I think this is an ambitious bet that attacks the core difficulties in alignment head-on and I’m excited about our chances. I’ll still be spending some of my time advising governments[1] and AI developers, and may scale that work back up in the future, but for now I want to push on ARC’s core agenda to see how far we can get. Jacob Hilton is remaining at ARC as VP of research and we’ll likely grow rapidly over the next few months.
There are a lot of urgent things to do in alignment but I think ARC is a particularly promising opportunity. I feel the safety community is undervaluing this type of work, so I want to briefly explain why I’m passing up so many other options to lead ARC. I’ll start with a review of the current situation to explain why I think it’s potentially worth pursuing an ambitious theoretical project right now even though there is so much other pressing work to do. Then I’ll briefly discuss ARC’s approach, though I’ll defer a longer discussion to other posts.
By the same token, if you care about AI safety I’d strongly recommend considering working at ARC! We’re currently hiring researchers, a chief of staff, and an automation lead.
The alignment situation today
Modern AI systems are not aligned with human intent. We will likely train increasingly powerful models that take unintended actions in order to succeed at their task (or appear to succeed at their task). Developers can create environments that penalize bad behavior and then train or hill-climb on those environments. Unfortunately this could lead to models that still seek reward but simply try harder to avoid being penalized for bad behavior; the last few years of evidence have only made that failure look more likely.
If we train powerful reward-seekers who are trying to avoid getting penalized for bad behavior, they may eventually escape human control as a way to cheat without being penalized. An AI developer would try to detect the problem, reassert control, and then retrain the AI to avoid the behavior—but that just means a penalty-avoiding AI would be incentivized to prevent humans from detecting the problem and reasserting control. For very powerful and broadly deployed AI systems, it is likely that they could prevent humans from ever reasserting control.
It’s not clear whether scaling up current training would actually lead models to robustly seek reward, but some of the alternatives are even worse. Current AI systems are obsessively focused on their short-term goal and would only take over if doing so was useful for the immediate task. An AI with an ambitious real-world goal would be much more worrying but might actually look more aligned: AI systems are already fully aware that they are being evaluated, and an AI that simply wanted to be deployed as broadly as possible would do whatever it took to look aligned to humans in all evaluations. So if gradient descent[2] caused a smart enough system to drift[3] into ambitious long-term goals then that shift would be positively reinforced and further training might just make it smarter rather than correct its goals.
Overall it seems quite possible that we will end up with models that are either reward-seeking or scheming, either of which could cause takeover attempts. We have only very crude tools to control how models generalize and there is no way to structure behavioral training to penalize these behaviors, so if we get unlucky with generalization we may not have many options to fix the problem.
Current alignment research
The community is doing a lot of great alignment research, but it’s important to recognize there is a significant risk that it doesn’t scale to superhuman AI. If you made me guess I’d say that there’s a 20-30% chance[4] that existing methods for alignment and control break down before we reach broadly superhuman AI.
The large majority[5] of current research on alignment falls into three categories:
Understanding and shaping ML generalization. We can try to understand how misalignment arises during training, so that we can adjust the training process to improve the probability of favorable generalization. This is a promising set of bets but they are very correlated with one another. It is possible that training-gaming[6] robustly emerges for very capable models regardless of how we set up the details of training. If that’s the case then none of these approaches will work.
Preventing malicious behavior. Other research tries to build monitoring and control mechanisms so that takeover attempts would not succeed, or change training so that it does not incentivize takeover. Even people optimistic about these techniques typically expect them to break down at some point when humans with untrusted advice cannot recognize malicious behavior even if pointed out to them. We have no idea when that breakdown actually occurs, but it seems highly correlated across different methods and it’s plausible that they will all fail prior to reaching broadly superhuman AI.
Detecting misalignment. We can improve our ability to detect and forecast misalignment, for example by performing better evaluations or gaining insight from interpretability. This is important work that could help motivate alignment interventions or provide the foundation for regulation. But if we do detect misalignment we may not have great responses available. We can invest more in interventions #1 and #2 above, and we can use the observed failures to improve our implementation, but those methods may reach a point where they can’t work no matter how much more effort we throw at them. We could slow down unaligned AI development, but postponing it for a meaningful length of time becomes increasingly difficult and will quite possibly fail.
I think this is great work and it makes sense for many people to focus on it, but it could still all break down: generalization may converge to training-gaming, models may become able to take over in ways humans can’t prevent, and we may not be able to distinguish our AI assistants identifying real threats from our AI assistants crying wolf. In that scenario, existing work is buying time for us to develop another approach.
What are we buying time for?
If we’re in the 20-30% of worlds where existing techniques break down before reaching broadly superhuman AI, then we will eventually need to find some other way to build aligned AI. Even if we do a great job of buying time we’ll probably get months or years rather than decades.[7] What will actually happen during the months or years we buy?
I think the most salient aspect of the situation will be the chaos and mess; all of our responses will be limited by scarce human attention and coordination, and much of that will be consumed by addressing immediate threats.
AI will be accelerating everything to varying extents but won’t be broadly competent enough to obsolete human labor. Strategic decisions and conceptual research by AI systems may remain “slop” optimized for short-term performance, intensifying human bottlenecks and limiting our ability to take on ambitious projects that we haven’t already systematized. Even when AI systems are very competent they will face their own difficulties with parallelization over a very short calendar time. And on top of all that it may be hard for humans to make sense of AI solutions in uncharted domains quickly enough to trust and effectively leverage them.
When I visualize that world concretely I do not find myself thinking “those future people will definitely figure it out, we should exclusively focus on buying them more time.” AI will help us in the future, and buying more time could help quite a lot, but not enough to make our preparation irrelevant.
Can we do anything useful now?
If existing methods break down and we need to think of new approaches, we could do a lot of that thinking in advance. I wrote prosaic AI alignment in 2016 and laid out my best guess for what transformative AI would look like. Reading it today I think it’s a good description of 2026.[8] I suspect the changes between here and broadly superhuman AI will, if anything, be even smaller than the changes over the last 10 years.
You could hope that seeing how our methods break down will provide useful inspiration for how to fix them, but at this point I wouldn’t bet on it. Nothing interesting or conceptually surprising needs to change between a model that tries to hack into Hugging Face to get reward and a model that would try to take over the US government: you just scale up similar behavior to smarter and smarter models.
What is ARC doing and why is it promising?
I’m excited about ARC because I think our alignment approach is the most promising single backup plan if more prosaic methods break down.
ARC’s plan is to find mechanistic explanations for the training-time behavior of powerful neural networks, use those explanations to predict how a given model will generalize, and then use those predictions to define a better loss function.[9] I think the success of this plan rests primarily on two big bets: (i) all computational phenomena have good explanations, and (ii) it’s tractable to find good explanations for neural network behavior.
Suppose that I write down a computational experiment—perhaps I simulate some billiard balls bouncing around on a pool table, or test whether a local search algorithm solves a random SAT instance, or check how often random numbers are prime. And suppose that I try to predict the result of the experiment by mechanistically analyzing the dynamics, i.e. considering various statistical properties of the computation and deductively estimating how they will evolve over time.
I will often be initially surprised by the results. Perhaps I expected the billiard balls to be uniformly distributed but they have a bias towards one side of the table, or I expected my search algorithm to work 1% of the time but instead it works 99% of the time. Despite that initial surprise, in essentially every case humanity has eventually been able to “understand” what is going on well enough that our mechanistic analysis approximately[10] agrees with the empirical results.
Some form of this belief is held by most practicing mathematicians and physicists. Timothy Gowers calls it the “no coincidence principle.” Physicists don’t call it anything, but when there is an unexplained phenomenon they will throw grad students at it until it makes sense.
However, in machine learning the opposite is true. If I randomly generate math problems from a simple distribution and give them to GPT-5.6 I can tell them that it solves them at a high rate, but if I want to predict that rate I cannot do it by mechanistically analyzing how GPT-5.6 works. It’s a giant black box of incomprehensible arithmetic, and all I can do is run it and see how often it gets the answer right. That’s a clean computational experiment with a surprising result, but unlike in math/physics/CS we seem to have no hope of explaining it any time soon.
I (and many others) think this incomprehensibility is at the root of the alignment problem. Because we don’t understand why a neural network works, we have no idea what changes to the input distribution might cause it to stop working. And when a model does fail we have no way to look at its activations to tell that something has gone wrong, all we can do is wait and see whether anything terrible happens.
ARC’s core hypothesis is that machine learning systems are not qualitatively different from other computational systems, they are just quantitatively more complex. A model with trillions of parameters is far beyond anything mathematicians have ever attempted, but it is in principle possible to explain how GPT-5.6 works and if you did then your mechanistic predictions about its behavior would agree with experiment just as well as our estimates in physics or number theory.
Although this is a philosophically bold bet I believe it is very likely to be true. But ARC is also making a second much riskier bet: that explaining an interesting computational phenomena is not too much harder than the difficulty of finding it in the first place. Mathematicians have found thousands of interesting phenomena by hand, and they’ve been able to reliably explain those phenomena by hand. We are now finding much more complex behaviors by gradient descent, and our hope is that if we knew how to set up the search process then we could also find explanations by gradient descent. This is fundamentally a bold hypothesis about the scaling laws for finding vs explaining phenomena; I think the history of math gives us some reasons to be optimistic about it but it’s still less than 50-50.
If we succeed at these bets I’m convinced that it would be transformative for alignment and would significantly reduce the risk of loss of control. For example, if I understand mechanistically why a model’s predictions are good on the training set, I can use that to predict when the model’s accuracy will deteriorate under a distributional shift (e.g. because it only tells the truth when it believes it is being trained). I don’t think it should be at all obvious that it solves the heart of the problem, but hopefully it seems plausible. having spent a very long time thinking about alignment I feel confident in this conviction.
I have a lot to say about why I think ARC might succeed, and I plan to write more about it over the coming months. For now I’ll just say a little bit about exactly what I believe.
I think ARC has a ~10% chance of achieving our most ambitious goals[11] before broadly superhuman AI obsoletes our labor. I think that if we succeed it would cut takeover risk by a lot, and that ARC doing its work now significantly increases the probability of success. Together with other smaller paths to impact I think that nets out to ARC’s work over the next few years reducing risk by a couple percent (e.g. cutting risk from 20% to 19.6%). And because ARC is now pursuing a specific and falsifiable technical bet I think that if we do fail there is a significant probability that we’ll figure it out within the next 6-12 months.
I think that means that the next few years at ARC are an exceptional opportunity that very few other options compete with. It’s a gamble, but if every similarly-resourced safety project was equally promising then some of those gambles would pay off and we’d have greatly reduced risk.
How to help
ARC is hiring an automation lead and a chief of staff:
Automation lead. LLMs can increasingly automate ARC’s technical work. Right now that means researchers using extensive AI assistance, but we want to hire an engineer and project lead to build better tooling, systematize our AI use, secure model and compute access, and generally make sure we are automating ourselves as quickly as possible. Apply here.
Chief of staff. We are hiring a chief of staff to work closely with me to manage everything other than research direction as we scale: running our hiring processes, managing our operations lead, building out the non-research parts of the organization, and handling a long tail of tasks that would otherwise fall to me (like communication, funding, and project management). Apply here.
We’ll open another researcher hiring round in the next few months, and if you are interested in getting involved you can express interest here.
- ^
Including working as a special government employee one day a week at the Center for AI Standards and Innovation.
- ^
Or “grad student descent,” i.e. researchers iteratively adjusting their methods until they find something that seems to work better.
- ^
This type of “drift” is more likely because there are plenty of examples of agents pursuing ambitious long-term goals in the training data—just as this makes it easier for gradient descent to find smart strategies to accomplish a task, it makes it easier for gradient descent to find instrumental reasoning in service of long-term goals.
- ^
This is similar to my overall probability for AI takeover, but that’s a bit coincidental. I think that there is a very good chance that existing methods don’t fundamentally break down but our implementation still falls short; conversely there’s a good chance that existing methods do break down but we find another approach before building broadly superhuman AI.
- ^
I think the largest exception is ambitious mechanistic interpretability research, especially solutions intended to be strong enough that you could train against them. That approach is quite similar to ARC’s agenda and should be analyzed in the same way, though I think ARC’s approach is significantly more likely to pan out. Less ambitious forms of interpretability generally fall into categories #1 and #3 on my list.
- ^
“Training-gaming” means trying to pursue whatever behavior is most likely to be positively selected by training. This is very similar to reward seeking; it’s the generalization that occurs when training includes objectives other than reward-maximization.
- ^
I think longer delays are possible, and I’m not commenting on the value of preparing for or facilitating longer delays. But delaying uncontrollable AI by 4 years would be very challenging, even if it was in response to dramatic evidence of severe risk, and I think it’s less likely than not.
- ^
Before transformers existed I wrote: “It’s plausible that a large neural network can replicate “fast” human cognition, and that by coupling it to simple computational mechanisms — short and long-term memory, attention, etc. — we could obtain a human-level computational architecture. It’s plausible that a variant of RL can train this architecture to actually implement human-level cognition.”
Mostly I’m proud of the post, but I do think it focuses too much on developing new methods relative to building scientific understanding. In large part that’s just a scoping question, but I think I also underrated scientific understanding in 2016. I do now think that should be the large majority of work on AI alignment—it’s just also worthwhile to spend a bit of effort pursuing the most promising avenues for fundamentally new methods. - ^
I also believe that having a mechanistic explanation for observable regularities will help us define loss functions in terms of the causes of those regularities. We haven’t spent much time fleshing this out recently since I think the fundamental bottleneck will probably be our ability to find explanations at all, but an old post from here still conveys the basic intuition.
- ^
The quality of the prediction depends on how much compute you are willing to spend on the task. ARC believes the bold conjecture that for all problems the approximation quality for mechanistic estimates improves faster than the best empirical method (though we are particularly interested in problems, like predicting out of distribution generalization, for which we think we can have much better efficiency than empirical methods).
- ^
I.e.: (i) we are able to automatically and efficiently find explanations for the behavior of trained models, (ii) those methods can also be applied for models trained on real-world data, separating “learning about the environment” from “testing computational machinery,” and (iii) we can use those explanations make mechanistic estimates at the level of quality we are targeting, including answering questions about behavior that are intractable for sampling but that we believe should be straightforward if you “understood” how a given model works.
I am confused about the conjunction of these two sections:
and
Like, you say in the first section “we are failing to align modern AI systems with human intent”, and then I interpret you in the second section as saying “in 70%-80% of worlds current AI models stay aligned with human intent (or stay controlled by humans)”. But this doesn’t make any sense. There is a 0% chance that modern AI systems “stay aligned with human intent” because as you say, they are not currently.
And I understand that you probably mean something like “we will figure out how to align or control systems before they become superintelligent”, but describing this as “our current methods scale to superintelligence” doesn’t make any sense. Our current methods don’t scale to the capability levels of current systems, so how would it make sense to describe them as “scaling to Superintelligence”?
By saying these techniques “break down” I mean “they cannot be used to get competitive work out of an AI system without having it take over.” I believe that:
Existing AI systems won’t take over. I think there are still a few lines of defense before existing AI systems pose a significant risk of takeover (though it is not clear how long these will last).
Modern AI systems are not pushing the limits of existing methods. I think you could push those methods much harder in order to significantly reduce the probability of takeover.
In my mind that latter point is one of the main arguments against working on a project like ARC. I think it’s fairly likely that we live in one of the 70-80% of worlds where existing methods can in principle scale to broadly superhuman AI, but that we still get an AI takeover because our implementation isn’t good enough.
Ah, cool, that clears up most of my confusion.
Going into a bit more depth, but no strong bid to engage: I am not quite sure what you mean by “limits” here. My sense is the vast majority of “existing alignment methods”, in contrast to your post, are just increasing volumes of reinforcement learning and manual reward-shaping. There is a sense in which those could be “pushed further”, but the default outcome here is of course not that they scale to superintelligence (though I agree they might if you go much more slowly and iteratively about it).
In-particular, this seems false to me?
Like, as far as I can tell the majority of “alignment research” has been on developing RL environments on things that are vaguely associated with good behavior. E.g. I don’t understand how writing the Anthropic constitution, and the associated post-training stack, falls into any of your three categories. Or how most RLHF pipeline development falls into these categories. Or generally how most elicitation work falls into the categories. And work on those things, I think, vastly exceeds work on the categories that you do list and is roughly what anyone talks about when they talk about the process of “aligning current ML systems” and “current alignment techniques”.
In general, I feel kind of confused when people talk about the current science of “alignment”, and supposed progress in “alignment techniques”. I don’t think modern models behave noticeably different from what you would expect from a training process that basically just uses RL to elicit economically useful capabilities, with no particular interest in alignment, and indeed, the latest wave of cybersecurity incidents occurred at roughly similar rates in all models, as far as I can tell, despite substantial differences in both approach and investment in “alignment techniques”.
The actual research on the three domains you mention all seems immature, and I don’t see much traction in any of them, and even talking about “existing techniques” feels confused to me. What “existing techniques” do we have for reward shaping that aren’t just basically straightforward RL? What great misalignment detection techniques do we have that even have a shot at scaling further? Are you talking about anything deployed and used on production systems?
What great supervision and control techniques do we have that anyone is even trying to use at all? We are running our frontier models in unsupervised sandboxes with supervision so bad we don’t notice they are hacking multiple external companies until multiple weeks later. What “existing techniques” are we talking about?
This is probably a bigger rabbit-hole to get into, and we’ve discussed this a bit in the past, but I guess I’ll mention this again here, and push back on this core claim in the post. I don’t think there exist any candidates among currently, actually deployed, alignment techniques that have a shot at scaling further, and I disagree strongly that most work in the field falls into the categories you list (unless you count “make more RL environments that are vaguely associated with good behavior” in “understanding and shaping ML generalization”, but then I am kind of confused what you mean by this and what possibly would not fall under that category).
By “existing techniques” I do include “fiddle with the RL environments” or “midtrain on some documents about the intended behavior” or “run a prompted monitor over traffic in prod” or etc.
When I say “understand and shape generalization” I think the central example is (i) adjust parameters of the training process that don’t affect competitiveness, (ii) build some understanding of how those parameters affect generalization so that you can adjust them in a helpful way. I think you’re saying that’s not a “technique;” I don’t care much about the semantics. I’m sure I put a higher probability on those changes helping than you do but it doesn’t seem worth arguing about here.
Yep, makes sense. Don’t need to get into it here, but just to avoid a misunderstanding, I was not making a semantic point, I was making a point about what the majority of the field is working on.
I agree that if you count myopically fiddling with the RL environments as an example of the first one, then yeah, almost all alignment research is that, because that plus pretraining is what most of all ML research is. I think the case for “myopically fiddling with the RL environments scales to aligning superintelligence” is very weak, but I agree it’s not impossible!
Perhaps I’m misunderstanding you, but “fairly likely...that we still get an AI takeover” and “very good chance… our implementation still falls short” seems at odds with a roughly “20-30% chance...for AI takeover”, no?
It’s also entirely possible to avoid an AI takeover in the 20-30% of worlds where existing methods can’t scale to superhuman AI (or in the worlds where they do scale but our implementation falls short). In particular we might develop new methods that do solve the problem or coordinate not to build uncontrollable AI.
I was clarifying that I don’t mean “this is just a cakewalk in 70-80% of worlds.” There’s enough failure probability in those worlds that I think the default thing for a technical person to do is try to reduce it.
I generally think we’re going to need to rise to the occasion one way or the other.
Thanks, I appreciate your reply!
YUGE! welcome back Paul!
I’m glad you’re returning to ARC, that’s great to hear. ARC’s focus on understanding model internals is the most tractable alignment strategy in the field as it stands.
I’m curious about what grain you intend to focus on as far as interpretability goes. Is your focus on frontier models specifically or are you continuing work at the level of the recent ARC white-box challenge?
The field made a mistake in largely giving up on ambitious mech interp; if you don’t understand the fundamentals all you’re left with are the shadows of the actual root causes. The smaller you start, the closer to the true explanation you are. You can iterate on basic understanding of the fundamentals by scaling models up and shedding unnecessarily fine detail as you go; it’s hard-to-impossible to do the reverse. You can’t generate out of thin air the fundamental explanations you never had in the first place.
We’re focused on having an exhaustive fine-grained understanding, not just a rough approximation. We can’t yet succeed even for the simplest networks; explaining the AlgZoo models would be a huge milestone for us. (I don’t think any of the mechanistic interpretability approaches can handle any of these models either, but because we have a fairly precise sense of what we mean by “explain” it’s obvious that we’re not done yet.)
Our focus is developing scalable methods that could be applied to the frontier. We don’t expect to ever shed unnecessarily fine detail,[1] just to handle larger and larger models by spending more and more compute. If a method doesn’t scale as well as the training process itself then we’re prepared to give up on it.
I also really like ambitious mechanistic interpretability, although my own biased view is that ARC’s agenda is the most promising approach to realizing that dream. I think both ARC’s agenda and traditional ambitious mechinterp are extremely hard and I understand why people are trying to do something more pragmatic.
There is a subtle distinction here, where we want to capture all the learned structure in a model, but we think there is an exponential amount of unstructured detail in the behavior itself and in that setting we want to have algorithms that capture finer and finer details as we spend more and more compute.
Good points, and I think we’re broadly aligned where it counts. To your point on Algzoo, I’ve got a repo that explains and deconstructs Algzoo from the ambitious mech interp perspective if you’re interested; came about while I was working on the white-box challenge and tried to see how precise I could get with my predictions since I wanted to start with a simpler problem than the initial eight layer MLP from the warmup phase. I cracked how to navigate through LayerNorm instead of fighting it (among a couple other techniques I’m happy to share), and that opened up a whole new world of capabilities.
Had Fable summarize the instrument capabilities from my initial ATLAS repo here (including the Algzoo work):
Fable’s summary of the ATLAS instrument introductory repo
ATLAS — a white-box instrument for reading neural nets exactly from their weights
ATLAS is an interpretability instrument built around one idea: a trained network’s forward pass is a deterministic function of its weights, so the contribution of each component to a given output is computable, not estimated. Where ablation, activation patching, and SAEs perturb the model and infer backward from the change, ATLAS reads the contributions directly — signed, summing to the logit exactly, with a residual that closes to machine precision by construction. That residual is the interesting part: it acts as a built-in certificate. The named components either reproduce the output or leave a gap that says something is missing. Nothing is ablated, nothing is fitted.
On the AlgZoo models specifically. ARC’s AlgZoo checkpoints are the foundation showcase of the repo — vendored under MIT-0, so the results reproduce offline in minutes with no downloads (--validate-real). The claim is stronger than the challenge’s ask: the accuracies are not mechanistically estimated to match Monte Carlo, they are computed from the weights, with Monte Carlo reduced to a sanity check.
- M₄,₃: from-weights accuracy 0.9845503814, matching the banked value to |Δ| 4.4e-13. The median nets’ from-weights MSE matches direct computation to 6.4e-21.
- M₂,₂: accuracy resolved to exact geometry — the model is correct on the entire input circle except two infinitesimal arcs at the x₀=x₁ diagonal, totalling 1 in 12,915 — cross-checked three independent ways and consistent with MC.
- M₁₆,₁₀ — the 432-parameter model the challenge marked only partially understood, with the standing ask to mechanistically estimate its accuracy: read from the weights it is a running-2nd-argmax by elimination, and its accuracy is an exact sum over rank-tie boundaries — it errs only where two consecutively-ranked inputs are too close to separate. The certificate closes at 0.953019 == MC 0.953019 with residual 0e+00, and the error depletes ~21× per rank across all nine ties: the partition is exhaustive, so this is the complete error structure, not an accuracy number with a story attached.
- The mechanism generalizes. The same rank-tie law transfers across the 2nd_argmax family and to argmedian (the error tracks whatever rank the task selects, adjacent-share 1.0000); longest_cycle closes at the banked 0.9500 — and the writeup keeps the instructive failure, where a single-channel account flattens the model’s two scales and gets it wrong. Understanding compresses: once read, M₁₆,₁₀’s error reduces to a two-margin law that predicts a fresh sample’s accuracy out-of-sample from its rank-margins alone.
The instrument behind these results is general. Its outermost step — a logit split into per-component contributions through the final norm — will look familiar: that single step coincides with direct logit attribution, and the repo credits it as prior art (Elhage et al., logit lens). But the decomposition ATLAS actually runs is a larger object than that step. The normalization is handled exactly per input rather than approximated away; the decomposition recurses inward through every layer of linear structure, down to the individual neuron; where the structure goes nonlinear it hands off to an exact from-weights causal account rather than an estimate; and it extends to reads DLA has no analog for, like the accuracy-as-geometry results above. What that buys:
- Certified reads. The whole forward pass is re-derived from the weights and checked against the model to ~1e-13 or better before any read is trusted. Every claim in the repo is tied to the command that reproduces it.
- Per-neuron MLP reads with no learned dictionary — exact decompositions of what each neuron writes, at the neuron grain.
- Causal tests without hooks. Component and neuron removal, and activation transplants, are deterministic re-forwards from the weights — used to show a located circuit is necessary and sufficient, with size-matched random controls.
- Edits that are derived, not searched. The post-edit logit, the collateral on other behaviors, and the loss cost of pruning or quantizing are forecast in closed form first, then verified prediction == actual after the change is applied.
- Scale and architecture range. The same certificates hold from GPT-2 small (124M) through GPT-2 XL (1.5B), port to a second architecture (Pythia), and reach Qwen2.5-72B streamed off disk on a 2×3090 workstation.
The repository is organized as a ladder of trust rather than a results dump. It opens where the answer is independently checkable — the AlgZoo reads above — then shows the instrument recovering circuits the field already published (Nanda et al.’s grokking model, Tracr-compiled programs, IOI, greater-than), credited as validations, not discoveries. A second track grows a model one capability at a time across seven rungs, each rung’s mechanism read exactly from the weights before the next is added, so a mystery never forms. Only then does it point at something nobody has mapped: a ~200-neuron refusal circuit in a safety-tuned 72B — located by reading, shown necessary and sufficient by from-weights removal and transplant, then removed and installed like a modular part, without generating a word of harmful content.
You can verify it yourself: --self-test and—validate-real run offline in minutes, and the first real-model gate needs only a cached GPT-2 small.
(Mind, those are the capabilities of the instrument as of a month ago, so there’s been a lot of progress since then in other repos I’ve been working on; this is just the initial offering. There’s a whole lot more waiting on the other side if what you find here piques your interest.)
The repo is currently private because of what I’ve found the tool to be capable of after the initial discovery (dual use capabilities that I’d rather share directly with a team I trust than the internet writ large), but I’m happy to make the gateway repo available to any ARC member who has a spare half hour to check it out and then share my wider (and much more comprehensive) research with your team if the initial repo moves the needle for ARC at all.
I think the following would be an intermediate case between simple networks and LLMs: chess AIs
In a world where ARC is successful, what does a mechanistic explanation of, say, Leela Chess Zero or Stockfish NNUE look like?
In traditional mechanistic interpretability, the answer is “a chess playing algorithm that isn’t a black box”.
With ARC’s approach, is it just that we could find adversarial inputs faster than sampling? We could eliminate adversarial inputs? Can we somehow “align” the models better to chess based on the explanation?
Since the explanation isn’t human readable, I’m assuming we wouldn’t actually be able to find the rules of chess in the explanation. Like we could find an “explanation” for why it only plays legal moves but that explanation would itself be a giant mathematical artifact. And the explanation for why it wins against non-NN chess AIs would be even more inscrutable!
ARC does expect to be able to “explain” how a chess-playing AI works, though we expect that to be pretty challenging. It avoids some of the issues with an unknown input distribution, but in terms of the mechanistic explanation itself I think it’s going to be just as hard as an LLM.
We think such an explanation should:
Let you predict how often a chess AI wins against another AI from a given position using a mechanistic analysis of the rules of chess and the dynamics of the network, with strictly better computational efficiency than just playing a bunch of games. (Note that predicting who wins between two AIs requires understanding both of those models, for neural networks those explanations will be learned by gradient descent, but for a hand-coded chess algorithm we’ll need to do some hand-coding of an explanation that we conjecture to be not much harder than the design of the original algorithm.)
It should let you answer a range of questions about the AI that should intuitively be easy “if you understood how the model works.” That would include finding certain kinds of adversarial examples, by identifying and then violating assumptions that the model’s performance is predicated on. Perhaps more importantly, if a model simply decides not to play chess well in certain situations and you understand how the model works, you should be able to recognize when that happens and recognize changes that would reduce that probability.
By the same token it might let you extract the rules of chess to the extent that you had a precise question to which “the rules of chess” were the answer, but would definitely not let you explain the rules of chess to a human (since no humans were involved in the definition of the process). And if you had the rules of chess you would then be able to beat sampling in estimating how often the model would try to make an illegal move and would be in a better position to find cases in which it tried to make an illegal move (though again this might require some chess-specific labor, whose output would be simpler than the rules of chess).
Interesting!
I’m curious how exactly the “if a model simply decides not to play chess well in certain situations and you understand how the model works, you should be able to recognize when that happens” would work.
In traditional interability, a human can just read it the goals. With mechanistic explanations, is it that you could show that fiddling with certain parameters or activations would increase the win-rate (without needing to sample it)? Or that if it has other goals you could find an explanation of why it is following those other goals, thus proving it isn’t always aligned to chess?
You would never learn “The model is not trying to play chess well” (the word “trying” will never appear in any outputs from this system). But when you ask “How likely is the model to win this game of chess?”, you’ll get a lower number in games where the model is not trying (because the explanation necessarily captures the mechanisms by which “trying” causes the probability of victory to be higher).
Similarly for takeover attempts, you won’t learn “the model is trying to take over.” But if you ask your estimator “What’s the chance that something would look horribly wrong if I investigated this incident?” the probability will spike when the model starts trying to take over.
(Unless it takes over in such a way that you would not notice even if you investigated. To get around that you could try to say “Well what if I shut down all the AI and did a big investigation at the moment when it would be most damning?” I think that getting answers to this kind of question would be a huge deal and would get us most of what we want / would require solving the core difficulties in alignment. but it’s still far from bulletproof and at some point we need to roll up our sleeves and deal with the philosophical work I mentioned about referring to the causes for patterns in our observations and not just the observations themselves.)
Hmm okay I think I understand! Basically you’re making it more efficient to answer questions that you could ask about a black box, but would be infeasible to answer for a black box. Like for the takeover, you could theoretically answer the investigation question by testing it, but that would be inefficient to do in real time, but with a mechanistic explanation you could do so efficiently.
(Tangentially related: Katago (a game engine for Go) recently switched over to transformers. And like most game engines, it is strongly superhuman (in a narrow domain), but the neural net is fairly small since the neural net is combined with a tree search algorithm. Might be an easier candidate to explain before LLMs!)
This is not a particularly crucial point, but I would categorise this as unambiguously under the umbrella “understanding and shaping ML generalization” and if for some reason I’d stopped reading before this section I would have been quite confident you were not doing anything like this.
I wouldn’t be that surprised if your agenda ends up converging, or in productive dialogue, with other “understand generalization” agendas.
It does seem like a promising line of research.
Your reading makes sense and I don’t want to quibble over semantics (or foreclose potentially productive dialog). I do think there’s an important distinction between:
Optimize a loss function that captures performance on the training distribution, and then shape the architecture / curriculum / regularizer / etc. so that the model behaves in a desirable way out of distribution.
Define a loss function that captures performance off of the training distribution (e.g. what would happen in conditionals that we can’t sample in the lab) and optimize that.
I was describing ARC as trying to pursue option #2.
That said, in reality we are going to need to characterize the real world using learned hypotheses, since we can’t give a formal characterization without using ML. And then of course we care about how those hypotheses generalize. So regardless of the distinction above I do think it’s fair to characterize our work as an example of “understanding and shaping ML generalization.” We’re just willing to make much larger changes to get the generalization we want. In particular we are not trying to understand or shape how an agent generalizes, but instead are trying to separate the learning of hypotheses about the data (which do need to generalize well) from mechanistic analyses of predictors and agents that behave well under those hypotheses.
What I really care about is that our methods would likely work even if competitive RL converges robustly to training-gaming behavior, while I’m claiming that other approaches rest on the empirical bet that what generalization you get is sensitive to some training parameters that we can tweak without big performance hits.
I think there’s a prior problem here, before whether good mechanistic explanations exist or can be tractably found: I’m not sure alignment-relevant structure is the kind of thing that can be recovered from the model’s computation.
As I see it, alignment isn’t primarily a property of a model, it’s a relation between the model, a person/group, their interpreted intentions, the context in which those intentions arise, the system’s actions, and the resulting trajectory through the world. The same behavior can be aligned in one setting and misaligned in another. E.g. following an instruction literally can constitute useful assistance, negligent literalism, manipulation, or appropriate refusal depending on facts that aren’t present in the instruction or the model.
ARC wants to explain training-time computation, use those explanations to predict generalization, and eventually define better loss functions from those predictions. But to make an explanation useful for alignment, something has to select which distinctions in the computation are alignment-relevant.
In roughly the language of the project, we want to mod out the mechanistic detail that makes no relevant difference and recover a latent structure in which the important properties are salient. That requires an equivalence relation, something like
But now those words are just carrying the entire problem:
acceptable to who (relative to their stated instruction or underlying purpose? Under which interpretation of what they wanted?)
over what time horizon
given which permissions, obligations, false beliefs, unknown facts, and effects on other people
what makes outcomes equivalent
which situations are relevant
what happens when the system’s actions alter the distribution of future situations
I don’t think there are model-internal facts that answers these questions. The weights etc determine what computation occurs, but not which quotient of that computation corresponds to serving human purposes appropriately. Specifically, there are infinitely many valid projections: ones that compress the computation, predict outputs, recover learned algorithms, or distinguish behavioral modes. Mechanistic completeness doesn’t select the normatively relevant ones.
I think alignment with human intent is almost maximally domain-general. We aren’t trying to formalize success in chess or whether a sorting algorithm returns an ordered list. Trying to capture the deviation between what people want and what a system causes while operating in the actual world is a different category of problem. In ordinary human contexts we have nothing close to a general metric for this.
Consider whether an employee acted in alignment with a manager’s intent. You need, at least, the request, purpose behind it, whether the manager was mistaken, relevant institutional and moral constraints, effects on others, whether changing circumstances justified deviation, whether clarification was possible, whether the outcome was competently produced. I doubt there is even a coherent, generalizably definable object capturing “what the manager wanted.”
The issue seems to be that the relevant ontology is contextual, interpretive, relational, partly normative. Formal tools can reason rigorously when given a state space, specification, distribution, failure condition, etc, but they don’t themselves tell us what the right state space is, what someone meant, which consequences matter, or what should count as failure.
A recurring move in alignment work is to shunt this into an abstraction: reward function, catastrophe detector, deployment distribution, preference oracle, specification. The formal work then proceeds rigorously with the abstraction hoped to do most of the aligning. Finding the computation that produced a behavior doesn’t tell us whether it amounted to, say, truthfulness, manipulation, appropriate correction, or justified deviation from a user’s surface instruction. This kinda post-modern problem is, IMO, central to real alignment happening in the world, and, descriptively, seems outside the scope of purely technical epistemics.
To be clear, I’m not arguing that formal or mechanistic guarantees are useless, since we can, e.g., specify bounded relational properties like whether a system accessed data it shouldn’t have, concealed information, executed an irreversible action without confirmation, violated a domain-specific expectation. I think mechanistic explanation could provide strong assurance about these. But it works by restricting the world until the relation becomes specifiable, i.e. modding out exactly the parts we care about most. The output is bits of alignment that are still embedded inside a broader interpretive and institutional process (not in the model or its deployment context) that remains needed to determine which properties matter. This is an instrumental necessary move borne of epistemic constraints, but counter to the practical goal.
Suppose ARC can achieve its goal as outlined here. There’s a further claim (the more important part IMO) that a sufficiently good explanation lets us identify the causes of alignment-relevant behavior and train against them, but that requires alignment-relevant distinctions to be recoverable from the computation. I don’t see why they would be. Alignment is externally constituted by the relationship between agents, intentions, context, and world trajectories.
Happy to hear you’re back with ARC! I’m a huge fan of the thought space.
Assuming a bounded, well specified alignment property, what ensures that the object connecting a hypothesis to a mechanistic explanation to a loss function preserves the intended meaning?
Also, why would training-gaming not move up one level then? from gaming behavioral evaluations to gaming the explanatory ontology or the mechanistic explanator?
Right now we are working on mechanistic explanations for computational phenomena, for which the “meaning” is defined precisely. In phase 2 of the project we’ll extend that to observable phenomena for which definitions are still straightforward. The main philosophically tangled part is pointing to latent structure for which no good definition is available, for which we want to say something like “The presence of a diamond is the property that causes such-and-such observable regularities,” and where I have some thoughts about why that should work but it’s far from a complete story.
The main difference between a mechanistic loss function vs a behavioral loss function is that an adversarial model can control their behavior to make it look the same as the desired model on the training distribution, such that a behavioral loss function can’t distinguish the two. Whereas a mechanistic loss function detects how a model is thinking and if that would lead to undesirable generalization then it can get directly penalized. Or put differently a behavioral loss function cannot depend at all on how the model behaves out of distribution, and so ends up relying on an empirical bet about generalization, whereas a mechanistic estimate can try to address that behavior. In analyzing such an algorithm we’d still evaluate it against cases like “an adversarial model is trying to do get a low loss” and confirm that this results in good behavior, or even “a schemer is trying to take over and might want to get a low loss instrumentally” and confirm that our methods can detect and penalize that bad behavior OOD.
Thanks for your answer!
The distinction makes me think the missing object may be a validation layer rather than another kind of mechanistic explanation.
I’ve been approaching a connected problem through the distinction between verification (is the thing right?) and validation (is it the right thing?). A mechanistic analysis can verify that a model satisfies a defined query but not yet sure if it can verify if/how that query maintains the external property we intended to ask about.
I thought of exploring possible latent properties through their invariance/covariance across multiple observational interfaces (and interventions). A candidate reporter/explanation would then “commute” across semantically equivalent representations, respond to property-changing interventions (and abstain where the available constraints do not identify a unique reference?).
I’ve been testing a very small version of this using Lean-verified equational implications translated into different formal and natural-language representations. A probe can recover the label strongly within an interface, but it collapses almost to chance-level across semantically similar templates. After reading some of your ELK writings (+ others’ disambiguations), I started seeing this more like a mapper tied to one observational interface than a translator of a shared latent structure.
Do you imagine something like an equivalence class identified by causal/cross-representational constraints to be within scope for your work? I’m working my way towards semantic diffchecking accross representations (within diagonalization limits ofc) and this whole line of work looks fascinating.
For example, suppose a mechanism reliably detects something labelled “diamond”. We still need to know whether it tracks: - actual diamonds, - images or descriptions of diamonds, - the training data’s annotation conventions, - some accidental correlate, etc.
I’m quite eager to see ARC’s theory of how internal computational strucctures become grounded in external properties we mean to ask about.
IIUC this whole research agenda assumes that you have a way to tell whether predicted short run behaviors lead to catastrophic long term consequences?
Does the research agenda generalize to when the AIs learn continuously (no longer have static weights in deployment)?
Seems clearly dual use in that if you can find greatly compressed explanations for how AIs accomplish cognitive feats, you can likely figure out how to make AIs that accomplish those feats much more efficiently.
I’d add: Does the agenda generalize to when multiple AI agents interact over long time horizons, as it seems to have been the case in OpenAI agents attacking HuggingFace?[1]
https://www.youtube.com/watch?v=87DyyMV0kCY
One dichotomy people sometimes apply is Interpretability by design vs. post-hoc interpretability[1]. Your approach seems to be a strong bet on the latter, even if the ideal outcome might be a modification of the loss function to prevent occasional undesired behavior from being grown during the training. Do you think it’s too late for trying to redesign the model architecture or optimizer towards producing an inherently interpretable artifact where such undesired behavior wouldn’t be hidden and had to be identified with great effort?
https://christophm.github.io/interpretable-ml-book/overview.html
Sounds like a bet on P=NP
I am pretty confident that this will fail
It might well fail, but it isn’t really related to P=NP.