Alex Gibson

• Alex Gibson 18 Sep 2026 9:33 UTC

  1 point

  0

  in reply to: leogao’s comment on: leogao’s Shortform

  In this strict adversarial sense then the answer is we’ve not interpreted any SGD trained neural networks I think.

  Perhaps modular addition but only because the input space is sufficiently small we can just brute force every input to be sure it’s not running a different program.

  Maybe it’d be possible to make progress if you specified some sufficiently high complexity level on the hidden program.

• Alex Gibson 17 Sep 2026 14:42 UTC

  6 points

  3

  in reply to: leogao’s comment on: leogao’s Shortform

  Indeed the takeaway of the paper is roughly speaking that there is a negative relationship between the length of proof required to prove that a network achieves a fixed level of accuracy, and the degree of mechanistic understanding.

  A brute force proof is incredibly long, because you have to give a computational trace of every possible input to the network (in the case, possible inputs). So you get as tight a bound as you can, but you have no mechanistic understanding (and indeed I’d say this is a “bad” explanation of the network).

  The current shorter proofs we have trade off some accuracy, but they are significantly shorter. These require you to engage with the mechanisms of the network, doing “interpretability”.

  Jason Gross has published a number of other papers in this vein. Relevant to the modular addition transformer mentioned above, attempting to find a non vacuous proof of performance of linear length in parameter count lead to Modular addition without black-boxes: Compressing explanations of MLPs that compute numerical integration, which gives a detailed mechanism for the MLP layer in the modular addition task.

• Alex Gibson 8 Sep 2026 22:54 UTC

  4 points

  0

  in reply to: leogao’s comment on: leogao’s Shortform

  Formally verify that an agent can’t escape from a sandbox.

• Alex Gibson 8 Sep 2026 12:43 UTC

  64 points

  36

  in reply to: Thane Ruthenis’s comment on: Thane Ruthe­nis’s Shortform

  I think it’s most likely just paranoia.

  It may just be that conditioning on making a successful attempt at NS building on current promising approaches, that is the approach most likely to be used. The approach is not as niche as it’s being made out to be.

  My guess would be that no information leaked to OpenAI apart from the mere fact that Navier Stokes was within reach of current frontier models. Given this information, OpenAI spins up a massive effort pouring lots of compute into the problem, and get a solution out.

• Alex Gibson 1 Sep 2026 22:00 UTC

  2 points

  1

  in reply to: Joseph Van Name’s comment on: Joseph Van Name’s Shortform

  Surely the relevant question is can you freeze a network and then have the network unable to be trained to interpret .

• Alex Gibson 28 Jul 2026 23:54 UTC

  1 point

  0

  in reply to: programjames’s comment on: leogao’s Shortform

  I have similar experience except I blame myself for being bad at communicating instead of the reader lol. The customer is always right etc etc.

• Alex Gibson 28 Jul 2026 21:42 UTC

  1 point

  0

  in reply to: Lucius Bushnaq’s comment on: Linda Linse­fors’s Shortform

  

  It might be easiest if we just discuss tokens that you take issue with here?

  If it helps I have a (very rough) draft from 1.5 years ago—Duplicate token neurons in the first layer of GPT-2 analysing the mechanism.

  What i’d say is that I think the neuron is probably not exactly monosemantic (it fires on “cons” always for instance). It’s just a good example of where the neuron’s behaviour is mostly explained by the duplicate pattern, but where this isn’t clear immediately.

  What the draft analyses is the contribution just from positional information to the neurons activation, ignoring token specific contributions.

• Alex Gibson 16 Jul 2026 10:07 UTC

  2 points

  0

  in reply to: Francisco Ferreira da Silva’s comment on: Com­pressed Com­pu­ta­tion un­der L⁴ Loss is likely Com­pu­ta­tion in Superposition

  Yes, that’s the scenario I was intending.

  Hmm I was viewing the undershooting of the ReLU as being a minor correction. After all it’s not like the expected value is taken to , it’s just weakly interpolated there. In either case we can always modify to undershoot or whatever is required by the design.

  Yeah leakage to inactive features is always the concern. MSE is pretty lenient about this—some when considering the scenario use to punish outliers. Like MSE doesn’t care very much if a few inactive features falsely activate with high magnitude as long as it’s only with probability. I guess helps here but would need to do the calculations properly.

  But yeah given linear bottleneck width in and random codes (unsure about designed ones) there’s an exponentially small w.r.t chance that a given codeword of length will have no collisions. So the construction is very unlikely to be exactly computing.

  All you gain is that the network can be layers long and you can solve the compressed computation setup for it, and the only thing that matters for MSE is the bounds on the final output. So you can imagine having a massive classifier network for instance which just outputs or at the end. And knowing this you truncate all the blocks to output values between and , to bound the error.

  By the way, have you tried higher values of , i.e: ? Do you end up with the same results? It would be kinda cool if you got a range of families of constructions for different values of .

• Alex Gibson 15 Jul 2026 23:24 UTC

  2 points

  0

  on: Com­pressed Com­pu­ta­tion un­der L⁴ Loss is likely Com­pu­ta­tion in Superposition

  Interesting result.

  Irrespective of training, the solution given in the post should theoretically work for fairly general as well, correct?

  Suppose , with inputs , , and . [If can modify so that it is in an arbitrarily small region around the origin, while staying in required form]

  Divide the network width into blocks of size .

  Have the th feature write to a subset of blocks (With corresponding to the binary codewords in the post). [i.e: Write to the blocks so that if there are no collisions between active codewords, will be the contents of the block.]

  Now apply the remainder of (apart from ) block-wise. And then read off as usual, while applying .

  The blocks in without collisions will have computed exactly. And then we just need errors to not blow up on collided blocks. If we are told that the final hidden layer of is bounded on the typical input distribution then can append a truncation layer to bound how disruptive collided blocks can be.

• Alex Gibson 15 Jul 2026 12:24 UTC

  LW: 8 AF: 5

  0

  AF

  in reply to: Linda Linsefors’s comment on: Linda Linse­fors’s Shortform

  Tokens which occurred earlier in context window. With distance from the average position of previous occurrences determining the strength (so tends to fire on when a token was repeated a while back).

  The way you find this neuron is by looking at the neurons which duplicate token heads feed into.

• Alex Gibson 15 Jul 2026 11:25 UTC

  LW: 3 AF: 2

  0

  AF

  in reply to: Linda Linsefors’s comment on: Linda Linse­fors’s Shortform

  To be clear, I would be surprised if more than 2-3% of MLP neurons were monosemantic. It’s just not so rare.

  A tricky thing is that it’s easy to verify monosemanticity, but ruling out monosemanticity is a lot harder. For example, the following neuron has a fairly simple pattern https://​​www.neuronpedia.org/​​gpt2-small/​​0/​​1168, but at first glance seems to fire at random.

• Alex Gibson 14 Jul 2026 23:09 UTC

  LW: 3 AF: 2

  0

  AF

  in reply to: Linda Linsefors’s comment on: Linda Linse­fors’s Shortform

  f”neuronpedia.org/​​gpt2-small/​​{layer}/​​{neuron}” is useful for getting a sense of what different LLM neurons are used for on a range of different texts. Unfortunately, I believe neuron-level dashboards were discontinued, as the link only works for GPT2-Small.

  I couldn’t give a percentage, but finding monosemantic neurons is fairly common as you scroll through the neurons in GPT2-Small. It’s easy to underestimate the density of monosemantic neurons because back when autointerp was being done on neurons, LLMs were a lot worse than they are now. My hot take is that it’s just way easier to spot visual patterns than to spot patterns in text, which is why visual neurons get their monosemantic reputation.

  A couple use cases for single ReLU neurons:

  Commonwealth English vs American English Neuron. The pre-activation of this neuron tracks something like the Naive Bayes log probability of the input using Commonwealth spellings versus American spellings. The threshold then selects for text which uses Commonwealth spellings. So the log probability is linearly accessible prior to the MLP, but prior to the MLP the model has no direction just to indicate it believes there are Commonwealth spellings.

  Single ReLU neurons can perform an AND over discrete(-ish) features (note AND is non-linear). Example: the Known bigram neuron fires on the second token of a bigram the model knows (has a bit of an America tilt, but it will fire on generic known bigrams).

  The mechanism is that Head 0.7 attends from the suffixes of known bigrams to their matching prefixes, copying a “valid prefix” direction into the residual stream; the current token supplies a “valid suffix” direction . The neuron thresholds on the sum and fires only when both and are present with appropriate strength. When the current token is a valid suffix but no matching prefix precedes it, Head 0.7′s attention typically spreads over tokens that lack , diluting the prefix signal below threshold; alone isn’t enough.

• Alex Gibson 8 Jul 2026 9:53 UTC

  9 points

  0

  in reply to: Lucius Bushnaq’s comment on: Ping pong com­pu­ta­tion in superposition

  One construction that occurred to me that’s making me doubt the problem setup a little at least for :

  You can abuse the linear circuit embedding associated with circuit to communicate arbitrary binary codes to the bottom stream provided that you assume the sum of the input coordinates is bounded away from (Analogous to the bloom filter scenario). So transmit a binary code encoding the weights of the circuit and you win by just writing a generic circuit emulator and lose the dependence on (Potentially need bits if you wanna encode output in superposition).

  I think this is obviously pretty silly because in a real network you’d have to store the bits associated with the circuits somewhere and it’s just a quirk of the precise setup.

  Wonder what you think of instead assuming smth like the residual stream takes the form of concatenated blocks of the form (input | dense binary code of length log(T) encoding the circuit). So making the formatting rigid to avoid encoding too much info in the weight matrices themselves.

  In general transmitting binary codes to the bottom is fairly useful though. In the setup of Compressed computation (https://​​www.lesswrong.com/​​posts/​​ZxFchCFJFcgysYsT9/​​compressed-computation-is-probably-not-computation-in), feature can transmit an overcomplete basis vector associated with feature , which can then be used to encode the output of feature . Can also transmit the set of active features via a Bloom Filter.

• Alex Gibson 7 Jul 2026 23:49 UTC

  5 points

  0

  in reply to: Joseph Miller’s comment on: Joseph Miller’s Shortform

  I find Anthropic’s style irritating (Mostly because Mech Interp ppl are not the intended audience of the posts ig), but reluctantly I found it to be a pretty interesting paper on the technical level.

  “The J-Lens is an improvement of the logit lens.”: The J-lens is answering a different question from the logit lens, so it’s not best described as just an improvement imo.

  The J-lens gives a global linear approximation for the impact that intervening will have on the outputs at future tokens, as opposed to just the output at the current token. This leads to qualitatively different tokens being surfaced in comparison to the logit lens, and their results here are pretty interesting.

  Agreed it’s definitely overhyped though, yeah. But that’s what Anthropic does! It’s probably not too bad a trade off in exchange for getting more people into the field (as long as people in field maintain appropriate skepticism).

  I do wish that Anthropic would publish a preprint alongside the blog post that could plausibly be submitted to conference, because I think it would be far more readable if they were forced to compress things down to the key experiments.

• Alex Gibson 6 Jul 2026 16:33 UTC

  1 point

  0

  in reply to: Lucius Bushnaq’s comment on: Ping pong com­pu­ta­tion in superposition

  Thank you :)

  I would be interested in hearing more about what you get out of the construction, because I am kinda obsessed with finding these sorts of toy networks at the moment, but am kind of unsure how to score the constructions.

  It seems like there’s a lot of fruit in picking a mathematically well defined task and optimising constructions, and this is my current focus (see Bloom Filter post as another example). Coming up with good tasks to optimise is hard though.

• Alex Gibson 22 May 2026 23:50 UTC

  2 points

  0

  on: Which tech­ni­cal AI safety fields are go­ing to be au­to­mated first?

  My model of things is that there is a core set of generally applicable research skills, and then there is topic specific technical knowledge required to conduct research in that particular area.

  I don’t think models are bottlenecked by technical knowledge. I think they just don’t currently have the general capability required to do broad exploratory research, outside of narrowly specified verifiable problems.

  I would guess all these subfields will get automated at the same time, once said capability is developed.

  Different groups have specific agendas, and within said agendas you can try to automate the production of incremental papers (which are important!) But in the limit you’re going to have to develop concepts beyond the current frame if you don’t want to stagnate, which requires general exploratory research ability.

• Alex Gibson 21 May 2026 8:25 UTC

  21 points

  1

  in reply to: Julian Bradshaw’s comment on: anaguma’s Shortform

  To be fair I think the idea of using algebraic number theory to approach the problem had been tried before (Tsimerman mentions he tried a similar approach that the model ultimately succeeded with, but didn’t persist with it.) It’s quite a general trick to use algebraic number theory for constructions in the plane, as you have the lattice associated with the ring of integers of number fields.

  I personally am blown away by the proof but it would be far more impressive had it come up with a novel connection between fields, or indeed if it had turned out there wasn’t a counterexample and it proved a tight upper bound (See Gowers’ initial reaction.)

• Alex Gibson 25 Jan 2026 11:35 UTC

  1 point

  0

  in reply to: leogao’s comment on: leogao’s Shortform

  There are surely reasons to do ambitious interp that are not the stated goal of ambitious interp?

• Alex Gibson 5 Jan 2026 14:39 UTC

  2 points

  0

  in reply to: Fabien Roger’s comment on: Ger­ard Boxo’s Shortform

  Anthropic practically entrapped Claude to get that behavioural evidence.

• Alex Gibson 5 Jan 2026 1:51 UTC

  2 points

  0

  in reply to: TristanTrim’s comment on: Se­man­tic Topolog­i­cal Spaces

  Good luck with it. I do think the broad direction is pretty promising.