Ok I have a neat construction for z=1, https://www.lesswrong.com/posts/g9uMJkcWj8jQDjybb/ping-pong-computation-in-superposition that works pretty well ( with width and layers), and zero error. Note that is exact here, not asymptotic.
I’m trying to find a similar construction that scales like in the number of parameters required, without just scaling the number of layers up. I’d also be curious if it’s possible to avoid scaling parameters linearly with , but it seems quite difficult.
The restriction of the loss to the target feels like cheating, to be honest. The linear model claim is scoped to reconstruction loss, where you genuinely don’t see superposition as far as i’m aware. And in this case, the reconstruction loss would be poor, because the vectors are nested so close to each other that adjacent features false fire.
I agree with the core point about finding alternative models of superposition though. As far as I know, there is no evidence that the Toy Model paper is accurate to how real models actually represent things, except at the broadest level. Towards Monosemanticity in fact notes divergence from the Toy Model paper (see Feature Splitting).
On the model itself, for , and , why can’t you place vectors equidistant around the circle, allowing for arbitrarily many features?
Do you have formal definitions of what exactly you mean by the input space, or what you mean by the output space? What are the underlying sets, and what topology are you equipping them with? Wouldn’t the output space just be the interval , and the input space ?
Reading the midterm, the actual question was about the distance between a non-empty compact and a non-empty closed set in . Which is indeed a pretty simple exercise.
My model of why SAEs work well for the Anthropic analysis is that the concepts discussed are genuinely ‘sparse’ features. Like predicting ‘Rabbit’ on the next line is a discrete decision, and so is of the form SAEs model for. We expect these SAE features to generalize OOD, because the model probably genuinely has these sparse directions.
Whereas for ‘contextual / vibes’ based features, the ground truth is not a sparse sum of discrete features. It’s a continuous summary of the text obtained by averaging representations over the sequence. In this case, SAEs exhibit feature splitting where they are able to model the continuous summary with sparser and sparser features by clustering texts from the dataset together in finer and finer divisions. This starts off canonical, but eventually the clusters you choose are not features of the model, but features of the dataset. And at this point the features are no longer robust OOD because they aren’t genuine internal model features, they are tiny clusters that emerge from the interaction between the model and the dataset.
So in theory the model might have a direction corresponding to ‘harmful intent’, but the SAEs split the dataset into so many chunks that to recover ‘harmful intent’ you need to combine lots of SAE latents together. And the OOD behaviour arises from the SAE latents being unfaithful to the ground truth, not the model having poor OOD behaviour. Like SAE latents might be sufficiently fine that you can patch together chunks of the dataset to fit the train dataset, in a non-robust way.
As for concepts that generalize OOD—I suppose it depends what is meant by OOD? Like is looking at a dataset the model wasn’t exposed to, but that it reasonably could have been, OOD? If so, the incentive for learning OOD robust concepts is that assuming that most text the model receives is novel, this text is OOD for the model, so if its concepts are only relevant to the text it has seen so far, it will perform poorly. You can also argue regularisation drives short description lengths, and thus generalising concepts. Whether a chunk of the training set is duplicates / similar is kind of irrelevant, because even if only 50% of text is novel, the novel text still provides the incentive for robust concepts.
If I understand correctly, inactive circuits mistakenly activating is the main failure mode—once this happens, things go downhill quickly. So the bottleneck is robustly knowing which circuits should be active.
Could we use O(d) redundant On-indicators per small circuit, instead of just 1, and apply the 2-ReLU trick to their average to increase resistance to noise?
In the Section 5 scenario, would it help to use additional neurons to encode the network’s best guess for the active circuits early on, before noise accumulates, and preserve this over layers? You could do something like track the circuits which have the most active neuron mass associated with them in the first layer of your constructed network (though this would need guarantees like circuits being relatively homogeneous in the norm of their activations).
The reason I think this is a reasonable idea is that LLMs do seem to compute binary indicators of when to use circuits, separate from the circuits themselves. Ferrando et al. found models have features for whether they recognize an entity, which gates fact lookup. In GPT2-Small, the entity recognition circuit is just a single first-layer neuron, while the fact-lookup circuit it controls is presumably very complex (GDM couldn’t reverse-engineer it). This suggests networks naturally learn simple early gating for complex downstream computations.
Interesting post, thanks.
I’m worried about placing Mechanistic Interpretability on a pedestal compared to simpler techniques like behavioural evals. This is because ironically, to an outsider, Mechanistic Interpretability is not very interpretable.
A lay outsider can easily understand the full context of published behavioural evaluations, and come to their own judgements on what the behavioural evaluation shows, and if they agree or disagree with the interpretation provided by the author. But if Anthropic publishes a MI blog post claiming to understand the circuits inside a model, the layperson has no way of evaluating those claims for themselves without diving deep into technical details.
This is an issue even if we trust that everyone working on safety at Anthropic has good intentions. Because people are subject to cognitive biases and can trick themselves into thinking they understand a system better than they actually do, especially when the techniques involved are sophisticated.
In an ideal world, yes, we would use Mechanistic Interpretability to formally prove that ASI isn’t going to kill us or whatever. But with bounded time (because of alternative existential risks conditioning on no ASI), this ambitious goal is unlikely to be achieved. Instead, MI research will likely only produce partial insights that risk creating a false sense of security resistant to critique by non-experts.
As a more productive question, say we had an LLM, which, amongst other things, if there is a known bigram encoded in the residual stream of the form (corresponding to known bigram ), potentially with interference from other aspects, outputs a consistent vector into the residual stream from an MLP layer. This is how GPT-2 encodes known bigrams, hence the relevance.
And say that there are quadratically many known bigrams as a function of hidden neuron size, so that in particular there are more bigrams than residual stream dimension. As far as I know, an appropriately randomly initialized network should be able to accomplish this task (or at least with random)
Is the goal for SPD to learn components for such that any given component only fires non-negligibly on a single bigram? Or is it ok if components fire on multiple different bigrams? I am trying to reason through how SPD would act in this case.
A definition of a subset of features i’m beginning to like is just “short sentence describing the input”
A model has a linear feature associated with a description if there is a direction in activation space such that when the model is run on an input[1], the resulting activation has a dot product above a threshold iff the short description is agreed to hold about the input (by a small set of biological neural networks).
This raises the question, what percentage of English descriptions of length words have linear features associated with them?
We want to rule out adversarial examples so in practice we just test for linear features on a fixed text corpus.
Anthropic practically entrapped Claude into blackmailing someone, and then a lot of mainstream news picked it up and reported it at face value. How are you going to escalate from that in the minds of a mainstream audience, in terms of behavioural evidence immediately legible to said audience? Have Claude set off a virtual nuke in a sandbox?
Good luck with it. I do think the broad direction is pretty promising.
I think even formally defining what you want the underlying set of ideal space to be would be a good post.
I personally find the informal ideas you discuss in between the topology slop ( sorry :) ) to be far more interesting.
The topology I’m more interested in I might call the “semantic topology” which would have as open sets any semantically related objects.
It sounds like you want to suppose the existence of a “semantic distance”, which satisfies all the usual metric space axioms, and then use the metric space topology. And you want this “semantic distance” to somehow correspond to whether humans consider two concepts to be semantically similar.
An issue if you use the euclidean topology on the output space [0,1]^2, and a “semantic topology” on the input space, is that your network won’t be continuous by default anymore. The inverse image of an open set would be open in the euclidean topology, but not necessarily open in the “semantic topology”. You could define the topology on the output space so that by definition the network is continuous (quotient topology) but then topology really is getting you nothing.
Am I right that the line of argument here is not about the generalization properties, but a claim about the quality of explanation, even on the restricted distribution? As in, we can use the fact that our explanation fails to generalize to the inputs (0,1) and (1,0) as a demonstration that the explanation is not mechanistically faithful, even on the restricted distribution?
Sometimes models learn mechanisms that hold with high probability over the input distribution, but where we can easily construct adversarial examples. So I think we want to allow explanations that only hold on narrow distributions, to explain typical case behaviour. But I think these explanations should come equipped with conditions on the input distribution for the explanation to hold. Like here, your causal model should have the explicit condition “x_1=x_2”.
Minimizing components used per input Minimizing number of global components
Lots of Mech Interp methods minimize the LHS of this equation. Circuit analysis such as IOI, SAEs, APD.
Optimizing the LHS is useful if you want to be able to “tell a story” about any input. In this case you want the fewest latents on any given input, so that your story on a particular input is not too long.
But there’s no reason to think that the model is actually using a sparse set of components /features on any given forward pass. A model might be using hundreds of different features, and combining components together in a harmony to produce the output. And if we don’t just want to tell a short story about an input, and want to understand faithfully what’s going on, then we are going to have to accept this. And we can’t just make a new feature for each of the exponential number of ways that components can be combined.
Just because a model is potentially combining lots of components on any given input does not make it intractable to understand at a high level. We know this because people have a good understanding of operating systems, even though for an operating system to complete a single task, it potentially has to use thousands of different parts of the system. But the global number of parts of the system is small enough for a human to comprehend, and any given forward pass of the operating system just looks like combining these well-understood parts.
I don’t care about prediction markets.
But people with the belief that we aren’t going to be able to fully understand models frequently take this as a reason not to pursue ambitious/rigorous interpretability. I thought that was the position you were taking, by using the market to decide whether the agenda is “good” or not.
I’m confused by this. The KL term we are looking at in the deterministic case is
DKL(P[X,Λ]||P[Λ]P[X1|Λ]P[X2|Λ]), right?
For simplicity, we imagine we have finite discrete spaces. Then this would blow up if P[X=(x1,x2),Λ=λ]≠0, and P[Λ=λ]P[X1=x1|Λ=λ]P[X2=x2|Λ=λ]=0. But this is impossible, because any of the terms in the product being 0 imply that P[X=(x1,x2),Λ=λ] is 0.
Intuitively, we construct an optimal code for encoding the distribution P[Λ]P[X1|Λ]P[X2|Λ], and the KL divergence measures how many more bits on average we need to encode a message than optimal, if the true distribution is given by P[X,Λ]. Issues occur when but the true distribution P[X,Λ] takes on values which never occur according to P[Λ]P[X1|Λ]P[X2|Λ], i.e: the optimal code doesn’t account for those values potentially occurring.
Potentially there are subtleties when we have continuous spaces. In any case I’d be grateful if you’re able to elaborate.