Alex Gibson

Karma: 118

Alex Gibson 25 Jan 2026 19:55 UTC
1 point
0
in reply to: leogao’s comment on: leogao’s Shortform
I don’t care about prediction markets.
But people with the belief that we aren’t going to be able to fully understand models frequently take this as a reason not to pursue ambitious/rigorous interpretability. I thought that was the position you were taking, by using the market to decide whether the agenda is “good” or not.

Alex Gibson 25 Jan 2026 11:35 UTC
1 point
0
in reply to: leogao’s comment on: leogao’s Shortform
There are surely reasons to do ambitious interp that are not the stated goal of ambitious interp? I doubt we will have a fully understandable model by 2028, but I still think the abstractions developed in the process will be helpful.
For instance, many of the higher-order methods like SAEs are based on assumptions about how activation space is structured. Studying smaller systems rigorously can give us the ground truth for how models construct their activation space, that can allow us to question/modify said assumptions.

Alex Gibson 13 Jan 2026 16:51 UTC
1 point
0
in reply to: Roman Belaire’s comment on: Tensor-Transformer Variants are Surprisingly Performant
At CE threshold chosen, capacity wouldn’t be the bottleneck.
Besides, MLP networks can store information as efficiently as the number of parameters they have.

Alex Gibson 9 Jan 2026 13:12 UTC
2 points
0
on: Alex Gibson’s Shortform
A definition of a subset of features i’m beginning to like is just “short sentence describing the input”
A model has a linear feature associated with a description if there is a direction in activation space such that when the model is run on an input^[1], the resulting activation has a dot product above a threshold iff the short description is agreed to hold about the input (by a small set of biological neural networks).
This raises the question, what percentage of English descriptions of length $n$ words have linear features associated with them?
1. ^
  We want to rule out adversarial examples so in practice we just test for linear features on a fixed text corpus.

Alex Gibson 5 Jan 2026 14:39 UTC
2 points
0
in reply to: Fabien Roger’s comment on: Gerard Boxo’s Shortform
Anthropic practically entrapped Claude into blackmailing someone, and then a lot of mainstream news picked it up and reported it at face value. How are you going to escalate from that in the minds of a mainstream audience, in terms of behavioural evidence immediately legible to said audience? Have Claude set off a virtual nuke in a sandbox?

Alex Gibson 5 Jan 2026 1:51 UTC
2 points
0
in reply to: TristanTrim’s comment on: Semantic Topological Spaces
Good luck with it. I do think the broad direction is pretty promising.

Alex Gibson 4 Jan 2026 22:53 UTC
2 points
0
in reply to: TristanTrim’s comment on: Semantic Topological Spaces
I think even formally defining what you want the underlying set of ideal space to be would be a good post.
I personally find the informal ideas you discuss in between the topology slop ( sorry :) ) to be far more interesting.
The topology I’m more interested in I might call the “semantic topology” which would have as open sets any semantically related objects.
It sounds like you want to suppose the existence of a “semantic distance”, which satisfies all the usual metric space axioms, and then use the metric space topology. And you want this “semantic distance” to somehow correspond to whether humans consider two concepts to be semantically similar.
An issue if you use the euclidean topology on the output space [0,1]^2, and a “semantic topology” on the input space, is that your network won’t be continuous by default anymore. The inverse image of an open set would be open in the euclidean topology, but not necessarily open in the “semantic topology”. You could define the topology on the output space so that by definition the network is continuous (quotient topology) but then topology really is getting you nothing.

Alex Gibson 4 Jan 2026 19:43 UTC
5 points
0
on: Semantic Topological Spaces
Do you have formal definitions of what exactly you mean by the input space, or what you mean by the output space? What are the underlying sets, and what topology are you equipping them with? Wouldn’t the output space just be the interval $[0, 1]$ , and the input space $[0, 1]^{N}$ ?

Alex Gibson 3 Jan 2026 17:44 UTC
8 points
0
in reply to: Linda Linsefors’s comment on: Rotations in Superposition
Ok I have a neat construction for z=1, https://www.lesswrong.com/posts/g9uMJkcWj8jQDjybb/ping-pong-computation-in-superposition that works pretty well ( $T = \frac{D^{2}}{d^{2}}$ with $D (1 + \frac{2}{d})$ width and $L + 3$ layers), and zero error. Note that $\frac{D^{2}}{d^{2}}$ is exact here, not asymptotic.
I’m trying to find a similar construction that scales like $z$ in the number of parameters required, without just scaling the number of layers up. I’d also be curious if it’s possible to avoid scaling parameters linearly with $z$ , but it seems quite difficult.

Alex Gibson 2 Jan 2026 20:38 UTC
4 points
2
in reply to: Pretentious Penguin’s comment on: College Was Not That Terrible Now That I’m Not That Crazy
Reading the midterm, the actual question was about the distance between a non-empty compact and a non-empty closed set in $R^{2}$ . Which is indeed a pretty simple exercise.

Alex Gibson 2 Jan 2026 17:41 UTC
1 point
0
on: Debunking claims about subquadratic attention
Text diffusion LLMs can be more efficient than autoregressive models in practice because it is usually more efficient on GPUs to do one big operation than many small operations in sequence, even when both require the same number of FLOPs^[1].
My mental model was that autoregressive models are slow during decoding because of the memory transfers required on each token. You need to swap the entire KV cache in and out of Global Memory on each token. I imagine this is what you mean here, just wanted to check this model is along the right lines?
I feel like there must be a way to allow for more flexibility in memory usage. There are clearly scenarios where what you want is MCMC style, constant memory / markov search, i.e: recall. In cases like this, autoregressive transformers are just painful because they inherently tie up iterations with memory usage. But there are scenarios where constant memory loses as well, i.e: sparse recall tasks.

Ping pong computation in superposition

Alex Gibson29 Dec 2025 16:31 UTC

13 points

0 comments3 min readLW link

Alex Gibson 26 Dec 2025 20:04 UTC
1 point
0
in reply to: Linda Linsefors’s comment on: Rotations in Superposition
We could allocate more of the total storage to on-indicators information and less to the rotated vector. Such a shift may be more optimal.
This is all I mean. Having the small circuits do error correction on on-indicator neurons is just a way of increasing that total percentage storage allocated to on-indicators in the larger network in an embedding agnostic manner. You can change your method for constructing the larger network later and this strategy would be orthogonal to such a change.
I think allocating a higher percentage to on-indicators should still apply even when adding cross-circuit computation.
Possibly there are some things you can eliminate right away? But I think often not. In the transformer architecture, at the start, the network just has the embedding vector for the first token and the positional embedding. After the first attention, the network has a bit more information, but not that much, the softmax will make sure the network just focus on a few previous words (right?). And every step of computation (including attention) will come with some noise, if superposition is involved.
The assumption “softmax will make sure the network just focus on a few previous words (right?)” is true for many attention heads, but not all of them. Some attention heads attend broadly across the whole sequence, aggregating many different tokens together to get the ‘gist’ of a text.
By the end of the first layer of GPT-2 Small, it has constructed a continuous linear summary of the previous $50$ tokens, and so has a Word2Vec style vector in its residual stream. So it knows that the text is about World War I, or about AI, that it is written in British English, that is is formal/informal, etc, all by the end of the first attention layer (before the first MLP). This is lots of information to go on in terms of turning off circuits etcetera (think Mixture-Of-Experts).
There is a regime where the (updated) framework works. See figure 8-11 for values T(d/D)^2 < 0.004. However for sizes of networks I can run on my laptop, that does not leave room for very much superposition.
Nice, ok. So asymptotically it works fine then? So the next step theory wise is having a framework that allows for cross-circuit computation, I guess.
Do you want to have a call some time in January?
There are probably lots of thing that isn’t explained as well as it could have been in the post
I would be grateful if you get the time. I have recently had some more ideas that are more useful potentially about this stuff that it might be worth discussing.

Alex Gibson 20 Dec 2025 14:08 UTC
LW: 1 AF: 1
0
AF
in reply to: Linda Linsefors’s comment on: Rotations in Superposition
To clarify where i’m coming from, I read the previous post as showing that assuming inactive circuits never activate due to superposition noise, we could get $T = O (\frac{D^{2}}{d^{2}})$ . This post shows that inactive circuits erroneously activate in practice, violating that assumption. I’m curious what asymptotics are possible if we remove this assumption and force ourselves to design the network to prevent such erroneous activations. I may be misinterpreting things, though.
If I understand your correctly, this is already what we are doing. Each on-indicator is distributed over $S$ pairs of neurons in the Large Network, where we used $S = 6$ for the results in this post.

Agnostic of the method for embedding the small circuits in the larger network, currently only $1$ out of $d$ neurons in each small network is being allocated to storing whether the small network is on or off. I’m suggesting increasing it to $c d$ for some small fixed $c$ , increasing the size of the small networks to $(1 + c) d$ neurons. In the rotation example, $d$ is so small that it doesn’t really make sense. But i’m just thinking about asymptotically. This should generalise straightforwardly to the “cross-circuit” computation case as well.
The idea is that while each of the $c d$ indicator neurons would be the same as each other in the smaller network, when embedded in the larger network, the noise each small network neuron (distributed across $S$ neurons in the large network) receives is hopefully independent.
If we assume that which circuits are active never changes across layers (which is true for the example in this post) there is another thing we can do. We can encode, the on-indicators, at the start, in superposition, and then just copy this information from layer to layer. This prevents the parts of the network that is responsible for on indicators from seizing. The reasons we didn’t do this is we wanted to test a method that could be used more generally.
This method also works under the assumptions specified in Section 5.2, right? Under Section 5.2 assumptions, it suffices to encode the circuits which are active on the first layer, of which there are at most $z$ . Even if you erroneously believe one of the $z$ circuits is active on a later layer, when it has turned off, the gain comes from eliminating the other $T - z$ inactive circuits. If the on-indicators don’t seize, then you can stop any part of the circuit from seizing in the Section 5.2 scenario.
I agree shared state/cross-circuit computation is an important thing to model, though. I guess that’s what you mean by more generally? In which case I misunderstood the post completely. I thought it was saying that the construction of the previous post ran into problems in practice. But it seems like you’re just saying, if we want this to work more generally, there are issues?
---------------------------------------
This series of posts is really useful, thank you! I have been thinking about it a lot for the past couple of days.

Alex Gibson 18 Dec 2025 21:34 UTC
LW: 3 AF: 2
0
AF
on: Rotations in Superposition
If I understand correctly, inactive circuits mistakenly activating is the main failure mode—once this happens, things go downhill quickly. So the bottleneck is robustly knowing which circuits should be active.
Could we use O(d) redundant On-indicators per small circuit, instead of just 1, and apply the 2-ReLU trick to their average to increase resistance to noise?
In the Section 5 scenario, would it help to use $T$ additional neurons to encode the network’s best guess for the active circuits early on, before noise accumulates, and preserve this over layers? You could do something like track the circuits which have the most active neuron mass associated with them in the first layer of your constructed network (though this would need guarantees like circuits being relatively homogeneous in the $L^{1}$ norm of their activations).
The reason I think this is a reasonable idea is that LLMs do seem to compute binary indicators of when to use circuits, separate from the circuits themselves. Ferrando et al. found models have features for whether they recognize an entity, which gates fact lookup. In GPT2-Small, the entity recognition circuit is just a single first-layer neuron, while the fact-lookup circuit it controls is presumably very complex (GDM couldn’t reverse-engineer it). This suggests networks naturally learn simple early gating for complex downstream computations.

Alex Gibson 6 Sep 2025 0:43 UTC
3 points
−1
on: Interpretability is the best path to alignment
Interesting post, thanks.
I’m worried about placing Mechanistic Interpretability on a pedestal compared to simpler techniques like behavioural evals. This is because ironically, to an outsider, Mechanistic Interpretability is not very interpretable.
A lay outsider can easily understand the full context of published behavioural evaluations, and come to their own judgements on what the behavioural evaluation shows, and if they agree or disagree with the interpretation provided by the author. But if Anthropic publishes a MI blog post claiming to understand the circuits inside a model, the layperson has no way of evaluating those claims for themselves without diving deep into technical details.
This is an issue even if we trust that everyone working on safety at Anthropic has good intentions. Because people are subject to cognitive biases and can trick themselves into thinking they understand a system better than they actually do, especially when the techniques involved are sophisticated.
In an ideal world, yes, we would use Mechanistic Interpretability to formally prove that ASI isn’t going to kill us or whatever. But with bounded time (because of alternative existential risks conditioning on no ASI), this ambitious goal is unlikely to be achieved. Instead, MI research will likely only produce partial insights that risk creating a false sense of security resistant to critique by non-experts.

Alex Gibson 11 Aug 2025 19:13 UTC
1 point
0
in reply to: zroe1’s comment on: Alternative Models of Superposition
I guess I mean cheating purely as “I don’t think this applies to to the Toy Model setting”, as opposed to saying it’s not a potentially valuable loss to study.
For p=1.0, I forgot that each of the noise features are random between 0 and 0.1, as opposed to fixed magnitude. The reason I brought it up is because if they were fixed magnitude 0.05 then they would all cancel out and face in the opposite direction to the target feature with magnitude 0.05. Now I reread the setting again I don’t think that’s relevant, though.

Alex Gibson 11 Aug 2025 18:44 UTC
8 points
1
on: Alternative Models of Superposition
The restriction of the loss to the target feels like cheating, to be honest. The linear model claim is scoped to reconstruction loss, where you genuinely don’t see superposition as far as i’m aware. And in this case, the reconstruction loss would be poor, because the vectors are nested so close to each other that adjacent features false fire.
I agree with the core point about finding alternative models of superposition though. As far as I know, there is no evidence that the Toy Model paper is accurate to how real models actually represent things, except at the broadest level. Towards Monosemanticity in fact notes divergence from the Toy Model paper (see Feature Splitting).
On the model itself, for $p = 0.0$ , and $p = 1.0$ , why can’t you place vectors equidistant around the circle, allowing for arbitrarily many features?

Alex Gibson 23 Jul 2025 18:02 UTC
9 points
2
in reply to: johnswentworth’s comment on: $500 Bounty Problem: Are (Approximately) Deterministic Natural Latents All You Need?
I’m confused by this. The KL term we are looking at in the deterministic case is
$D_{KL} (P [X, Λ] | | P [Λ] P [X_{1} | Λ] P [X_{2} | Λ])$ , right?
For simplicity, we imagine we have finite discrete spaces. Then this would blow up if $P [X = (x_{1}, x_{2}), Λ = λ] \neq 0$ , and $P [Λ = λ] P [X_{1} = x_{1} | Λ = λ] P [X_{2} = x_{2} | Λ = λ] = 0$ . But this is impossible, because any of the terms in the product being 0 imply that $P [X = (x_{1}, x_{2}), Λ = λ]$ is $0$ .
Intuitively, we construct an optimal code for encoding the distribution $P [Λ] P [X_{1} | Λ] P [X_{2} | Λ]$ , and the KL divergence measures how many more bits on average we need to encode a message than optimal, if the true distribution is given by $P [X, Λ]$ . Issues occur when but the true distribution $P [X, Λ]$ takes on values which never occur according to $P [Λ] P [X_{1} | Λ] P [X_{2} | Λ]$ , i.e: the optimal code doesn’t account for those values potentially occurring.
Potentially there are subtleties when we have continuous spaces. In any case I’d be grateful if you’re able to elaborate.

Alex Gibson 6 Jul 2025 15:59 UTC
2 points
0
on: Small foundational puzzle for causal theories of mechanistic interpretability
Am I right that the line of argument here is not about the generalization properties, but a claim about the quality of explanation, even on the restricted distribution? As in, we can use the fact that our explanation fails to generalize to the inputs (0,1) and (1,0) as a demonstration that the explanation is not mechanistically faithful, even on the restricted distribution?
Sometimes models learn mechanisms that hold with high probability over the input distribution, but where we can easily construct adversarial examples. So I think we want to allow explanations that only hold on narrow distributions, to explain typical case behaviour. But I think these explanations should come equipped with conditions on the input distribution for the explanation to hold. Like here, your causal model $M^{*}$ should have the explicit condition “x_1=x_2”.

Alex Gibson

Ping pong com­pu­ta­tion in superposition

Ping pong computation in superposition