Neuroscientist turned Interpretability Researcher. Starting Simplex, an AI Safety Research Org.
Adam Shai
Karma: 1,442
My thoughts on this are pretty uncertain but here’s some semi hot takes. Listed as they come to me in no particular attempt to make a coherent case for anything.
* We want ambitious but also good research. Easier said than done obviously.
* Having an ambitious research program is anti-correlated with experience. This is too bad, because research experience is extremely important and likely undervalued or hard to value (as is the case with the ever elusive taste, which I think one can develop through experience).
* Ambitious research is not worth much without all the other things that make good research good.
* It’s not obvious to me the issue is primarily one of incentives.
* The main thing one should do to get more people doing ambitious research is to do good ambitious research yourself, and to show by example what good ambitious research looks like, and convince others through object level things that this is a good path to take.
* It’s harder to do good ambitious research than it is to do good research that’s less ambitious. That’s a real tradeoff!
* I think funding opportunities are there for ambitious research to happen, but the bar is higher for it to be funded, and that seems reasonable?
* I’m still unclear on what people actually mean when they say ambitious research. For instance, sometimes “ambitious interp” is used for things I don’t personally find all that ambitious. I have no doubt that things I think are ambitious, others find quite pedestrian. Maybe it’s a taste thing.
What should I read to get caught up with the SOTA understanding?
Why do you think finding the true features should make the network look sparse and modular?
This is a self review. It’s been about 600 days since this was posted and I’m still happy and proud about this post. In terms of what I view as the important message to the readership, the main thing is introducing a framework and way of thinking that connects what is a pretty fuzzy notion of “world model” to the concrete internal structure of neural networks. It does this in a way that is both theoretically clear and amenable to experiments. It provides a way to think about representations in transformers in a general sense, that is quite different than the normal tactic taken in interpretability. One interesting thing given how interp has progressed since then, is that it provides a way to think about geometric structures in the activations of these networks, which is something the field seems to be moving towards (albeit not guided by theory the way this work is).
I spent a lot of time/effort on making the presentation both easy to follow while actually containing the important lessons. I feel this post does a great job at that. (and in fact, others around me were telling me I was taking too long and that I just needed to hit submit—I’m glad I didn’t follow that advice :)This work has been expanded on in a number of directions (a recent post summarizes three of those direction). We now have Simplex, an entire org that would likely not exist in the way that it does without this post (we got seed funding right before this post, but the ability to fundraise and attract talent were highly dependent on this post)! We are ~10 people and expanding. Starting this org, watching it grow, being part of the building up of a new way to think about/do interpretability has been the greatest intellectual achievement of my life, and again, I do not think this would have occured (at least not the way it did), without this post.
Some of the things we are working on is extending the framework to more complicated (factorizable) generative structures, which gives way to both a very nice story explaining how it is neural nets can cram so much into their activations, and the corresponding geometric relationship to computational structure. We also have a team dedicated to red-teaming the theory and application to experiments in a number of ways, one of which is more precisely testing the distinction between representing next-token vs. far future representaitons. We are also looking for the types of geometric structures our theory predicts in LLMs, and building tools to do unsuprvised finding of these geometric structures in LLMs.
In retrospect there are a few things I would have emphasized or done differently. The most concrete one would have been to include experiments on RRXOR in this writeup, which more directly test/show representations that have information beyond the next token, and that are predicted by the theory.Another regret is that I haven’t done a good job of keeping up with the public facing side of our work since this post. We made one post about our work since then, and it really does hide a lot of the insights we’ve made internally. This is not great, since I do believe that we have refined our way of thinking about representations and computations in these systems, in a way that the interp community needs.
More personally (and perhaps this isn’t relevant for a review since it really is just about me), this post came after switching fields from neuroscience, where I was for a decade, to interpretability. There was a lot of uncertainty and stress about that move, and quite a bit of wondering if I could actually contribute. So it was a big deal for me to see the community react positively to this work. Additionally, I had been interested in computational mechanics for a decade from within neuroscience, since it felt like it could say something about computation in biological neural nets, but I could never figure out how to get it to do anything useful. So it’s also a privilege to see the dream play out and be realized in neural nets, and to start a research org dedicated to that vision!
The start of the research program presented here is in a real sense a bet, in that it could fail to be useful at scale, for a few different reasons. But the fact that we are building up with both theory and experiment in a way where each informs the other, and that we were able to show the community a small part of what this looks like, is a lesson I hope the readership takes, even beyond the specific content of this post. (hot take) I think a lot of great work happens within this community, but too often the more theoretical and the more empirical approaches don’t talk to eachother as much as they should.
Pre-registering a prediction for experiment results testing why and when and how attention heads share information. We (simplex) will train transformers on data generated from the cartesian product of sequences generated from 2 independent Mess3 processes. If the degenerate eigenvalue of each is the same and positive (e.g. lambda=0.7 for both) then a transformer with a single attention head will learn it, and the attention patterns will be 0.7^(d-s) where d-s is the number of context positions between the source and desitination. If instead one of the processes has lambda = 0.7 and another lambda = 0.3, then this will seperate info. between two heads, where one has attention pattersn of 0.7^(d-s) and the other 0.3^(d-s). If instead we have one process with lambda=0.7 and the other with lambda = −0.3, then this will require 3 heads, with .7^(d-s) and the 0.3^(d-s) seperating between two heads to account for the fact that attention has to be positive. BUT if we have one process with lambda = 0.7 and the other with lambda = −0.7, then this will require 2 heads! With the 0.7^(d-s) being able to be shared on one head for the lambda=0.7 process, and for the −0.7 process when d-s is even, and then the odd d-s cases will be on the other head.
This looks great! Thanks for making the videos public.
Any chance the homework assignments/experiments can be made public?
A fun side note, that probably isn’t useful—I think if you shuffle the data across neurons (which effectively gets rid of the covariance amongst neurons), and then do linear regression, you will get theta_t.
This is a somewhat common technique in neuroscience analysis when studying correlation structure in neural reps and separability.
Somewhat related, in our recent preprint we showed how Bayesian updates work over quantum generators of stochastic processes. It’s a different setup than the one you show here, but does give a generalization of Bayes to the quantum and even post-quantum setting. We also show that the quantum (and post-quantum) Bayesian belief states are what transformers and other neural nets learn to represent during pre-training. This happens because to predict the future of a sequence optimaly given some history of that sequence, the best you can do is to perform Bayes on the hidden latent states of the (sometimes quantum) generator of the data.
Beautiful post, thank you!
In some sense I agree but I do think it’s more nuanced than that in practice. Once you add in cross-entropy loss on next token prediction, alongside causal masking, you really do get a strong sense of “operating on sequences”. This is because next token prediction is fundamentally sequential in nature in that the entire task is to make use of the correlational structure of sequences of data in order to predict future sequences.
There’s actually a market that’s even more directly relevant!
Simplex Progress Report—July 2025
This all sounds very reasonable to me! Thanks for the response. I agree that we are likely quite aligned about a lot of these issues.
Thanks for writing out your thoughts on this! I agree with a lot of the motivations and big picture thinking outlined in this post. I have a number of disagreements as well, and some questions:
It’s unfortunate that mech interp inherits the CNC paradigm, because despite many years of research, turns out it’s really hard to do computational science on brains, so computational neuroscience hasn’t made a huge amount of progress.I strongly agree with this, and I hope more people in mech. interp. become aware of this. I would actually emphasize that in my opinion it’s not just that it’s hard to do computational science on brains, but that we don’t have the right framework. Some weak evidence for this is exactly that we have an intelligent system that has existed for a few years now where experiments and analyses are easy to do, and we can see how far we’ve gotten with the CNC approach.
My main point of confusion with this post has to do with Parameter Decomposition as a new paradigm. I haven’t thought about this technique much, but on a first reading it doesn’t sound all that different from what you call the second wave paradigm, just replacing activations with parameters. For instance, I think I could take most of the last few sections of this post and rewrite it to make the point. Just for fun I’ll try this out here, trying to argue for a new paradigm called “Activation Decomposition”. (just to be super clear I don’t think this is a new paradigm!)
You wrote:
Parameter Decomposition makes some different foundational assumptions than used by the Second-Wave.One of these assumptions arises because Parameter Decomposition offers a clear definition of ‘feature’ that hitherto eluded Second-Wave Mech Interp. What Second-Wave Mech Interp calls a ‘feature’ can be defined as ‘properties of the input that activate particular mechanisms’. Notably, ‘features’ are here defined with reference to mechanisms, which is great, because ‘mechanisms’ has a specific formal definition!
This re-definition implies a difference in one of the foundational assumptions of Second-Wave Mech Interp that ‘features are the fundamental unit of neural network’. Parameter Decomposition rejects this idea and contends that ‘mechanisms are the fundamental unit of neural networks’.
and I’ll rewrite that here, putting my changes in bold:
Activation Decomposition makes some different foundational assumptions than used by the Second-Wave.One of these assumptions arises because Activation Decomposition offers a clear definition of ‘feature’ that hitherto eluded Second-Wave Mech Interp. What Second-Wave Mech Interp calls a ‘feature’ can be defined as ‘properties of the input that activate particular mechanisms’. Notably, ‘features’ are here [in Activation Decomposition] defined with reference to mechanisms [which are circuits of linearly decomposed activations], which is great, because ‘mechanisms’ has a specific formal definition!
This re-definition implies a difference in one of the foundational assumptions of Second-Wave Mech Interp that ‘features are the fundamental unit of neural network’. Activation Decomposition rejects this idea and contends that ‘mechanisms are the fundamental unit of neural networks’.
Perhaps a simpler way to say my thought is, isn’t the current paradigm largely decomposing activations? If that’s the case why is decomposing parameters so fundamentally different?
I think maybe one thing that might be going on here is that people have been quite sloppy (though I think it’s totally excusable and arguably even a good idea to be sloppy about these particular things given the current state of our understanding!), with words like feature, representation, computation, circuit, etc. Like I think when someone writes “features are the fundamental unit of neural networks” they are often meaning something closer to “representations are the fundamental unit of neural networks” or maybe something closer to “SAE latents are the fundamental unit of neural networks” and importantly, an implicit “and representations are only really representations if they are mechanistically relevant.” Which is why you see interventions of various types in current paradigm mech interp papers.
Some work in this early period was quite neuroscience-adjacent; as a result, despite being extremely mechanistic in flavour, some of this work may have been somewhat overlooked e.g. Sussillo and Barak (2013).
This is a nitpick, and I don’t think any of your main points rests on this, but I think the main reason this work was not used in any type of artificial neural network interp work at that time was that it is fundamentally only applicable to recurrent systems, and probably impossible to apply to e.g. standard convolutional networks. It’s not even straightforward to apply to a lot of the types of recurrent systems used in AI today (to the extent they are even used), but probably one could push on that a bit with some effort.
As a final question, I am wondering what you think the implications for what people should be doing are if mech interp is or is not pre-paradigmatic? Is there a difference between mech interp being in a not-so-great-paradigm vs. pre-paradigmatic in terms of what your median researcher should be thinking/doing/spending time on? Or is this just an intellectually interesting thing to think about. I am guessing that when a lot of people say that mech interp is pre paradigmatic they really mean something closer to “mech interp doesn’t have a useful/good/perfect paradigm right now”. But I’m also not sure if there’s anything here beyond semantics.
After thinking about this a bit more, the main point I’d want to make is less about recursive self improvement and more that just there’s a lot more capability in these models than people realize.
Whether that capacity is enough for recursive self improvement is another question that I’m not certain about either way but I think it’s at least plausible that it might be. I will note that humanity improves in its knowledge and capability without architectural change. That’s a rough analogy to the type of improvement I’m imagining.
i’m starting to think recursive self improvement is basically already possible with LLMs, even without anymore training ever. I’m pretty shocked with how much better my coding LLMs have become just by taking care to give the LLMs the right meta-context and information systems. I feel like I’ve moved from prompting, to figuring out what context is needed in addition to the prompt, to spending a bunch of time/effort building a knowledge structure so that the LLM can figure out its own context to get whatever done, and thats moved me from having LLMs write functions and scripts to large multi-file chunks of entire repositories. And I’m continually having the thought that’s like “ok but now I’m building this knowledge system that it can traverse and decide its own relevant context, but why can’t the LLM do that too? what would i need to setup for it to do that?” and i’m starting to feel like that’s a never ending thing.
Thanks for writing this! I have been thinking about many of the issues in your Why Won’t Interpretability Be Reliable section lately, and mostly agree that this is the state of affairs. I often think of this from the perspective of the field of neuroscience. My experience there (in the subsection of neuro research that I believe is the most analogous to mech interp) is that these are basically the same fundamental issues that keep the field from progress (though not the only reasons).
Many in the interpretability field seem to (implicitly) think that if you took neuroscience and made access to neural activities a lot easier, and the ability to arbitrarily intervene on the system, and the ability to easily run a lot more experiments, then all of neuroscience would be solved. From that set of beliefs if follows that because neural networks don’t have these issues, mech interp will have the ability to more or less apply the current neuroscience approach to neural networks and “figure it all out.” While these points about ease of experiments and access to internals are important differences between neuro. research and mech. interp., I do not think they get past the fundamental issues. In other words—Mech. interp. has more to learn from neuroscience failures than its successes (public post/rant coming soon!).
Seeing this post from you makes me positively update about the ability of interp. to contribute to AI Safety—it’s important we see clearly the power and weaknesses of our approaches. A big failure mode I worry about is being overconfident that our interp. methods are able to catch everything, and then making decisions based on that overconfidence. One thing to do about such a worry is to put serious effort into understanding the limits of our approaches. This of course does happen to some degree already (e.g. there’s been a bunch of stress testing of SAEs from various places lately), which is great! I hope when decisions are made about safety/deployment/etc., that the lessons we’ve learned from those types of studies are internalized and brought to bear, alongside the positives about what our methods do let us know/monitor/control, and that serious effort continues to be made to understand what our approaches miss.
Can a Finite-State Fox Catch a Markov Mouse? for more details
why I shouldn’t waste my time chasing this.
\
Some reasons that come to mind very quickly:
- Patch clamp experiments usually take place in slices with artificial cerebrospinal fluid (ACSF). The ephys properties can vary widely based on the experimental prep (angle that slice was taken, the temperature, the specific recipe used for the ACSF, the quality of the patcher, etc. etc.even if patching worked really well and was robust and reliable, the ephys properties at the soma (where vast majority of patching willt ake place) hardly describe the ephys of the entire dendritic tree, which is very complicated and space dependent, and incredibly nonlinear and variable.
Thanks for writing this up Owain!
I’ve been thinking about out of context reasoning lately and wanted to check if my thinking is in line with yours.
For the cases with induction and SFT, the way I’ve been thinking about it is that the model is somehow updating latent structure from its internal world model. So for example, we can conceptualize something like the function
f(x) = x // 4as a latent parent node, and then the observed(x, y)pairs consistent with it as child observations. You train on the(x, y)pairs, and the model infers the latent parent/rule that explains them. I actually thought I remembered seeing a DAG like this in one of your papers or slides, but I can’t find it now, so maybe I hallucinated it.This can be thought of like a kind of amortization of the inference over that DAG into the weights. Thinking of OOCR in this way makes me think it’s very important, since it seems to point at internal world models in a pretty concrete way.
But this seems different in kind to multi-hop reasoning, which until I read this post I had not thought about as being OOCR. In that case, it feels more like the model is chaining together already-known facts in the forward pass (importantly, with the intermediate step not written into the context). Is there also some non-tortuous way to think about that phenomenon as inference of a latent like in the induction case? Or do you think these are actually pretty different types of things, but still, both are OOCR?