Thanks Scott
Exciting! I look forward to the first “interesting circuit entirely derived by causal scrubbing” paper
I believe this post is (for the most part) accurate and demonstrates understanding of what is going on with logical induction. Thanks for writing (and coding) it!
OP came to mind while reading “Building A Virtual Machine inside ChatGPT”:
...We can chat with this Assistant chatbot, locked inside the alt-internet attached to a virtual machine, all inside ChatGPT’s imagination. Assistant, deep down inside this rabbit hole, can correctly explain us what Artificial Intelligence is.
It shows that ChatGPT understands that at the URL where we find ChatGPT, a large language model such as itself might be found. It correctly makes the inference that it should therefore reply to these questions like it would itself, as it is itself a large language model assistant too.
At this point, only one thing remains to be done.
Indeed, we can also build a virtual machine, inside the Assistant chatbot, on the alt-internet, from a virtual machine, within ChatGPT’s imagination.
Quick self-review:
Yep, I still endorse this post. I remember it fondly because it was really fun to write and read. I still marvel at how nicely the prediction worked out for me (predicting correctly before seeing the data that power/weight ratio was the key metric for forecasting when planes would be invented). My main regret is that I fell for the pendulum rocket fallacy and so picked an example that inadvertently contradicted, rather than illustrated, the point I wanted to make! I still think the point overall is solid but I do actually think this embarrassment made me take somewhat more seriously the “we are missing important insights” hypothesis. Sometimes you don’t know what you don’t know.
I still see lots of people making claims about the efficiency and mysteriousness of the brain to justify longer timelines. Frustratingly I usually can’t tell from their offhand remarks whether they are using the bogus arguments I criticize in this post, or whether they have something more sophisticated and legit in mind. I’d have to interrogate them further, and probably get them to read this post, to find out, and in conversation there usually isn’t time or energy to do that.
FWIW, I found the Strawberry Appendix especially helpful for understanding how this approach to ELK could solve (some form of) outer alignment.
Other readers, consider looking at the appendix even if you don’t feel like you fully understand the main body of the post!
in particular how much causal scrubbing can be turned into an exploratory tool to find circuits rather than just to verify them
I’d like to flag that this has been pretty easy to do—for instance, this process can look like resample ablating different nodes of the computational graph (eg each attention head/MLP), finding the nodes that when ablated most impact the model’s performance and are hence important, and then recursively searching for nodes that are relevant to the current set of important nodes by ablating nodes upstream to each important node.
First, no, the AGI is not going to “employ complex heuristics to ever-better approximate optimal hypotheses update”. The AGI is going to be based on an algorithm which, as a mathematical fact (if not proved then at least conjectured), converges to the correct hypothesis with high probability. Just like we can prove that e.g. SVMs converge to the optimal hypothesis in the respective class, or that particular RL algorithms for small MDPs converge to the correct hypothesis (assuming realizability).
Second, there’s the issue of non-cartesian attacks (“hacking the computer”). Assuming that the core computing unit is not powerful enough to mount a non-cartesian attack on its own, such attacks can arguably be regarded as detrimental side-effects of running computations on the envelope. My hope is that we can shape the prior about such side-effects in some informed way (e.g. the vast majority of programs won’t hack the computer) s.t. we still have approximate learnability (i.e. the system is not too afraid to run computations) without misspecification (i.e. the system is not overconfident about the safety of running computations). The more effort we put into hardening the system, the easier it should be to find such a sweet spot.
Third, I hope that the agreement solution will completely rule out any undesirable hypothesis, because we will have an actual theorem that guarantees it. What are the exact assumptions going to be and what needs to be done to make sure these assumptions hold is work for the future, ofc.
Thanks for the clarification! If I’m understanding correctly, you’re saying that the important part is decomposing activations (linearly?) and that there’s nothing really baked in about what a component can and cannot be. You normally focus on components, but this can also fully encompass the features as directions frame, by just saying that “the activation component in that direction” is a feature?
Nice summary! One small nitpick:
> In the features framing, we don’t necessarily assume that features are aligned with circuit components (eg, they could be arbitrary directions in neuron space), while in the subgraph framing we focus on components and don’t need to show that the components correspond to featuresThis feels slightly misleading. In practice, we often do claim that sub-components correspond to features. We can “rewrite” our model into an equivalent form that better reflects the computation it’s performing. For example, if we claim that a certain direction in an MLP’s output is important, we could rewrite the single MLP node as the sum of the MLP output in the direction + the residual term. Then, we could make claims about the direction we pointed out and also claim that the residual term is unimportant.
The important point is that we are allowed to rewrite our model however we want as long as the rewrite is equivalent.
Really excited to see this come out! I’m in generally very excited to see work trying to make mechanistic interpretability more rigorous/coherent/paradigmatic, and think causal scrubbing is a pretty cool idea, though have some concerns that it sets the bar too high for something being a legit circuit. The part that feels most conceptually elegant to me is the idea that an interpretability hypothesis allows certain inputs to be equivalent for getting a certain answer (and the null hypothesis says that no inputs are equivalent), and then the recursive algorithm to zoom in and ask which inputs should be equivalent on a particular component.
I’m excited to see how this plays out at REMIX, in particular how much causal scrubbing can be turned into an exploratory tool to find circuits rather than just to verify them (and also how often well-meaning people can find false positives).
This sequence is pretty long, so if it helps people, here’s a summary of causal scrubbing I wrote for a mechanistic interpretability glossary that I’m writing (please let me know if anything in here is inaccurate)
Redwood Research have suggested that the right way to think about circuits is actually to think of the model as a computational graph. In a transformer, nodes are components of the model, ie attention heads and neurons (in MLP layers), and edges between nodes are the part of input to the later node that comes from the output of the previous node. Within this framework, a circuit is a computational subgraph—a subset of nodes and a subset of the edges between them that is sufficient for doing the relevant computation.
The key facts about transformer that make this framework work is that the output of each layer is the sum of the output of each component, and the input to each layer (the residual stream) is the sum of the output of every previous layer and thus the sum of the output of every previous component.
Note: This means that there is an edge into a component from every component in earlier layers
And because the inputs are the sum of the output of each component, we can often cleanly consider subsets of nodes and edges—this is linear and it’s easy to see the effect of adding and removing terms.
The differences with the above framing are somewhat subtle:
In the features framing, we don’t necessarily assume that features are aligned with circuit components (eg, they could be arbitrary directions in neuron space), while in the subgraph framing we focus on components and don’t need to show that the components correspond to features
It’s less obvious how to think about an attention head as “representing a feature”—in some intuitive sense heads are “larger” than neurons—eg their output space lies in a rank d_head subspace, rather than just being a direction. The subgraph framing side-steps this.
Causal scrubbing: An algorithm being developed by Redwood Research that tries to create an automated metric for deciding whether a computational subgraph corresponds to a circuit.
(The following is my attempt at a summary—if you get confused, go check out their 100 page doc…)
The exact algorithm is pretty involved and convoluted, but the key idea is to think of an interpretability hypothesis as saying which parts of a model don’t matter for a computation.
The null hypothesis is that everything matters (ie, the state of knowing nothing about a model).
Let’s take the running example of an induction circuit, which predicts repeated subsequences. We take a sequence … A B … A (A, B arbitrary tokens) and output B as the next token. Our hypothesis is that this is done by a previous token head, which notices that A1 is before B, and then an induction head, which looks from the destination token A2 to source tokens who’s previous token is A (ie B), and predicts that the value of whatever token it’s looking at (ie B) will come next.
If a part of a model doesn’t matter, we should be able to change it without changing the model output. Their favoured tool for doing this is a random ablation, ie replacing the output of that model component with its output on a different, randomly chosen input. (See later for motivation).
The next step is that we can be specific about which parts of the input matter for each relevant component.
So, eg, we should be able to replace the output of the previous token head with any sequence with an A in that position, if we think that that’s all it depends on. And this sequence can be different from the input sequence that the input head sees, so long as the first A token agrees.
There are various ways to make this even more specific that they discuss, eg separately editing the key, value and query inputs to a head.
The final step is to take a metric for circuit quality—they use the expected loss recovered, ie “what fraction of the expected loss on the subproblem we’re studying does our scrubbed circuit recover, compared to the original model with no edits”
The genre of plans that I’d recommend to groups currently pushing the capabilities frontier is: aim for a pivotal act that’s selected for being (to the best of your knowledge) the easiest-to-align action that suffices to end the acute risk period.
FYI, I think there’s a huge difference between “I think humanity needs to aim for a pivotal act” and “I recommend to groups pushing the capabilities frontier forward to aim for pivotal act”. I think pivotal acts require massive amounts of good judgement to do right, and, like, I think capabilities researchers have generally demonstrated pretty bad judgment by, um, being capabilities researchers.
The genre of plans that I’d recommend to groups currently pushing the capabilities frontier is: aim for a pivotal act that’s selected for being (to the best of your knowledge) the easiest-to-align action that suffices to end the acute risk period. Per Eliezer on Arbital, the “easiest-to-align” condition probably means that you want the act that requires minimal cognitive abilities, out of the set of acts that suffice to prevent the world from being destroyed:
In the context of AI alignment, the “Principle of Minimality” or “Principle of Least Everything” says that when we are building the first sufficiently advanced Artificial Intelligence, we are operating in an extremely dangerous context in which building a marginally more powerful AI is marginally more dangerous. The first AGI ever built should therefore execute the least dangerous plan for preventing immediately following AGIs from destroying the world six months later. Furthermore, the least dangerous plan is not the plan that seems to contain the fewest material actions that seem risky in a conventional sense, but rather the plan that requires the least dangerous cognition from the AGI executing it. Similarly, inside the AGI itself, if a class of thought seems dangerous but necessary to execute sometimes, we want to execute the fewest possible instances of that class of thought.
E.g., if we think it’s a dangerous kind of event for the AGI to ask “How can I achieve this end using strategies from across every possible domain?” then we might want a design where most routine operations only search for strategies within a particular domain, and events where the AI searches across all known domains are rarer and visible to the programmers. Processing a goal that can recruit subgoals across every domain would be a dangerous event, albeit a necessary one, and therefore we want to do less of it within the AI (and require positive permission for all such cases and then require operators to validate the results before proceeding).
Ideas that inherit from this principle include the general notion of Task-directed AGI, taskishness, and mild optimization.
Having a plan for alignment, deployment, etc. of AGI is (on my model) crucial for orgs that are trying to build AGI.
MIRI itself isn’t pushing the AI capabilities frontier, but we are trying to do whatever seems likeliest to make the long-term future go well, and our guess is that the best way to do this is “make progress on figuring out AI alignment”. So I can separately answer the question “what’s MIRI’s organizational plan for solving alignment?”
My answer to that question is: we don’t currently have one. Nate and Eliezer are currently doing a lot of sharing of their models, while keeping an eye out for hopeful-seeming ideas.
If an alignment idea strikes us as having even a tiny scrap of hope, and isn’t already funding-saturated, then we’re making sure it gets funded. We don’t care whether that happens at MIRI versus elsewhere — we’re just seeking to maximize the amount of good work that’s happening in the world (insofar as money can help with that), and trying to bring about the existence of a research ecosystem that contains a wide variety of different moonshots and speculative ideas that are targeted at the core difficulties of alignment (described in the AGI Ruin and sharp left turn write-ups).
If an idea seems to have a significant amount of hope, and not just a tiny scrap — either at a glance, or after being worked on for a while by others and bearing surprisingly promising fruit — then I expect that MIRI will make that our new organizational focus, go all-in, and pour everything we have into helping with it as much we can. (E.g., we went all-in on our 2017-2020 research directions, before concluding in late 2020 that these were progressing too slowly to still have significant hope, though they might still meet the “tiny scrap of hope” bar.)
None of the research directions we’re aware of currently meet our “significant amount of hope” bar, but several things meet the “tiny scrap of hope” bar, so we’re continuing to keep an eye out and support others’ work, while not going all-in on any one approach.
Various researchers at MIRI are pursuing research pathways as they see fit, though (as mentioned) none currently seem promising enough to MIRI’s research leadership to make us want to put lots of eggs in those baskets or narrowly focus the org’s attention on those directions. We just think they’re worth funding at all, given how important alignment is and how little of an idea the world has about how to make progress; and MIRI is as good a place as any to host this work.
Scott Garrabrant and Abram Demski wrote the Embedded Agency sequence as their own take on the “Agent Foundations” problems, and they and other MIRI researchers have continued to do work over the years on problems related to EA / AF, though MIRI as a whole diversified away from the Agent Foundations agenda years ago. (AFAIK Scott sees “Embedded Agency” less as a discrete agenda, and more as a cluster of related problems/confusions that bear various relations to different parts of the alignment problem.)
(Caveat: I had input from some other MIRI staff in writing the above, but I’m speaking from my own models above, not trying to perfectly capture the view of anyone else at MIRI.)
Bah! :D It’s sad to hear he’s updated away from ambitions value learning towards corrigiblity-like targets. Eliezer’s second-hand argument sounds circular to me; suppose that corrigibility as we’d recognize it isn’t a natural abstraction—then generic AIs wouldn’t use it to align child agents (instead doing something like value learning, or something even more direct), and so there wouldn’t be a bunch of human-independent examples, so it wouldn’t show up as a natural abstraction to those AIs.
So indeed with cross-entropy loss I see two plateaus! Here’s rank 2:
(note that I’ve offset the loss to so that equality of Z and C is zero loss)
I have trouble getting rank 10 to find the zero-loss solution:
But the phenomenology at full rank is unchanged:
Does this mean that you expect we will be able to build advanced AI that doesn’t become an expected utility maximizer?
When talking about whether some physical system “is a utility maximizer”, the key questions are “utility over what variables?”, “in what model do those variables live?”, and “with respect to what measuring stick?”. My guess is that a corrigible AI will be a utility maximizer over something, but maybe not over the AI-operator interface itself? I’m still highly uncertain what that type-signature will look like, but there’s a lot of degrees of freedom to work with.
Do you think the current interpretability approaches will basically get us there or will we need qualitatively different methods?
We’ll need qualitatively different methods. But that’s not new; interpretability researchers already come up with qualitatively new methods pretty regularly.
Has the FTX fiasco impacted your expectation of us-in-the-future having enough money=compute to do the latter?
Basically no.
I’d like to make a case that Do What I Mean will potentially turn out to be the better target than corrigibility/value learning. …
I basically buy your argument, though there’s still the question of how safe a target DWIM is.
Still on the “figure out agency and train up an aligned AGI unilaterally” path?
“Train up an AGI unilaterally” doesn’t quite carve my plans at the joints.
One of the most common ways I see people fail to have any effect at all is to think in terms of “we”. They come up with plans which “we” could follow, for some “we” which is not in fact going to follow that plan. And then they take political-flavored actions which symbolically promote the plan, but are not in fact going to result in “we” implementing the plan. (And also, usually, the “we” in question is too dysfunctional as a group to implement the plan even if all the individuals wanted to, because that is how approximately 100% of organizations of more than 10 people operate.) In cognitive terms, the plan is pretending that lots of other peoples’ actions are choosable/controllable, when in fact those other peoples’ actions are not choosable/controllable, at least relative to the planner’s actual capabilities.
The simplest and most robust counter to this failure mode is to always make unilateral plans.
But to counter the failure mode, plans don’t need to be completely unilateral. They can involve other people doing things which those other people will actually predictably do. So, for instance, maybe I’ll write a paper about natural abstractions in hopes of nerd-sniping some complex systems theorists to further develop the theory. That’s fine; the actions which I need to counterfact over in order for that plan to work are actions which I can in fact take unilaterally (i.e. write a paper). Other than that, I’m just relying on other people acting in ways in which they’ll predictably act anyway.
Point is: in order for a plan to be a “real plan” (as opposed to e.g. a fabricated option, or a de-facto applause light), all of the actions which the plan treats as “under the planner’s control” must be actions which can be taken unilaterally. Any non-unilateral actions need to be things which we actually expect people to do by default, not things we wish they would do.
Coming back to the question: my plans certainly do not live in some childrens’ fantasy world where one or more major AI labs magically become the least-dysfunctional multiple-hundred-person organizations on the planet, and then we all build an aligned AGI via the magic of Friendship and Cooperation. The realistic assumption is that large organizations are mostly carried wherever the memetic waves drift. Now, the memetic waves may drift in a good direction—if e.g. the field of alignment does indeed converge to a paradigm around decoding the internal language of nets and expressing our targets in that language, then there’s a strong chance the major labs follow that tide, and do a lot of useful work. And I do unilaterally have nonzero ability to steer that memetic drift—for instance, by creating public knowledge of various useful lines of alignment research converging, or by training lots of competent people.
That’s the sort of non-unilaterality which I’m fine having in my plans: relying on other people to behave in realistic ways, conditional on me doing things which I can actually unilaterally do.
This post is on a very important topic: how could we scale ideas about value extrapolation or avoiding goal misgeneralisation… all the way up to superintelligence? As such, its ideas are very worth exploring and getting to grips to. It’s a very important idea.
However, the post itself is not brilliantly written, and is more of “idea of a potential approach” than a well crafted theory post. I hope to be able to revisit it at some point soon, but haven’t been able to find or make the time, yet.