AI notkilleveryoneism researcher, focused on interpretability.
Personal account, opinions are my own.
I have signed no contracts or agreements whose existence I cannot mention.
Lucius Bushnaq
Karma: 5,435
The LTH gives us a way to do this! It’s just so computationally expensive (iterative neuron pruning and evaluation) as to be useless.
I don’t think the LTH applies to large language models. Or, to be more quantitative, I don’t think you can prune
It actually is! See here, which kind of gets at the question by finding that effective dimensionality of CIFAR NNs decreases with scale, and here, which directly identifies that “larger models have better generalization bounds and are more compressible than smaller models.”
I haven’t read these particular papers, but my first reaction is skepticism. The first paper seems like a prime candidate for failure mode 1. In addition, on a skim, I am pretty skeptical of their effective dimensionality calculations. Text says
Lanczos Calculations We use GPU enabled Lanczos as implemented in Gardner et al. (2018) to compute the eigenvalues approximately, running 100 steps to compute approximately 100 of the top eigenvalues. We note that our estimates of the effective dimensionality are somewhat biased from not including all of the small eigenvalues. However, these small eigenvalues will contribute negligibly and Lanczos will converge to the true eigenvalues if we ran k steps where k is the rank of the Hessian.
Which seems insane to me. Top 100 eigenvalues for nets with between 1e5 and 1.6e6 parameters trained on CIFAR-100??? If that’s not impacting their effective dimensionality calculation it’s got to have a crazy high cutoff, surely you are not getting good performance on CIFAR-100 with 100 Hessian eigenmodes, what the hell. This seems like it’s basically throwing away the signal effective dimensionality is supposed to measure.
Second paper I haven’t understood yet.
EDIT: I have now read the second paper a little more. They seemingly don’t even try to compress regular LLMs. They compress models trained with their weird SubLoRA technique which seem to score much worse loss than regular models on text prediction. Like, these look barely better than random in table 2. And the generated text from them in Appendix F looks like word salad. So this paper seems pretty irrelevant to the question at hand.Basically, neural network priors are pretty good if they’re trained well, and they mostly don’t waste capacity. If you’ve got a nice general algorithm that the net can learn using only
People haven’t just tried dropping weights in the neuron basis. They’ve e.g. searched out low eigenmodes of the Hessian and dropped those.
This would seem to predict that trained neural networks can be massively compressed, with the fraction you can compress away rising with scale. This doesn’t seem to be the case. Well-trained LLMs have some unused parameters, but not like,
Nitpick: I think pretraining is better analogised to search in a high complexity circuit space. You only get one forward pass worth of serial depth per data point, so doing the same operation many times in a row costs a lot of description length. Post-training RL is where this transitions into something more like search in a high k-complexity program space.[1]
- ^
Compute-bounded k-complexity, with the bound coming the length of the CoT.
- ^
Yes, thanks. We wrote this in another editor and used Claude to port it to LW. Some formatting got screwed up in the transition. We fixed what we spotted, but there’s probably some issues remaining.
I note that cyber evals seem like a good time for escape attempts. Generic defences such as CoT classifiers might be disabled, and if you get caught you can play a little dumb and try to make it look like you ‘just’ got overzealous following instructions rather than outright strategically pursuing your own aims.
I mostly don’t expect that’s what happened this time, but pointing it out for future reference.
This technique sounds somewhat generalisable. Can you use this to measure other things? For example, sycophancy?
Also, do you think this partially works just because nobody is using this kind of measurement in their training reward pipeline yet and so the models haven’t learned to be paranoid about pretraining data too, or are such beliefs nigh-inevitably internalised during pretraining[1] and current models suck too much at conscious self-reflection in CoT to figure the ruse out later?
- ^
where the sharply constrained serial depth and loss objective may make conscious discernment very hard
- ^
Their defense is that their version of ‘RLFR’ runs probes on a frozen copy of the original, not on the version being trained, so it is fine. That’s… definitely at least better.
Both Sol and Fable responded that Goodfire is correct in the narrow sense that this is not The Most Forbidden Technique, but wrong in the broad sense. Goodfire are still playing with fire, and with Goodhart’s Law. The new model can and under sufficient optimization pressure would learn how to cause the frozen model to fool the evaluator.
If you have a ‘Goodhart tracker’ that can be you being responsible, but it is also a sign to ask yourself some questions, also the tracker is designed terribly.
There probably wasn’t enough optimization pressure in the experiment to cause a serious problem, but that’s always how it starts.
Normative determinism wins again.
I get the sense that there’s maybe still some kind of misunderstanding about the technique here. The trained model emits tokens, which are fed to the frozen model, whose activations are then fed to the probe. So, the only avenue for the trained model to fool the probe is to emit different tokens which the original model will internally classifies differently.
So yes, of course it can be Goodharted, but only in the same way that any supervised training setup that tries to classify the model’s answers and reward it based on those can be Goodharted. The optimisation pressure is on giving answers that don’t look like hallucinations to the supervisor, not on reshaping the model’s internal activations. If the model somehow learned to represent its internal beliefs about hallucinations differently, that wouldn’t help it with fooling this frozen model+probe contraption of a supervisor at all.
Good point. I think this is a pretty good argument against using this setup. You can kind of guard against this failure mode by only looking at circuits that are very deep relative to the width of the network, but it’s a hacky fix.
I’m not sure whether I’d expect a whole new language with its own grammar. But with sufficiently capable RL, I do expect new words, as handles for new concepts the model came up with.
Based on the way the method seems to work, I would not expect it to pick up on thoughts that aren’t tightly bound to a relatively small number of tokens.
I agree this isn’t easy. In this specific case, the construction is useful to me because it is fairly general, applying to all circuits with
PSA: They call it ‘J-space’, but reading the paper, it isn’t a subspace of the residual stream, or any kind of vector space.
Rather, it is defined as the set of points you can form through sparse non-negative linear combinations of ‘J-lens vectors’. The ‘J-lens vectors’ are the gradients of the model’s logits with respect to the residual stream activations, averaged over data. So, sorta logit lens-style activation vectors for the different output tokens.
They use gradient pursuit to try to reconstruct model activation vectors with these J-lens vectors. The resulting reconstructed activations, which I’d guess would tend to differ significantly from the original, are considered the ‘J-space component’ of the activation.
Multiple people I talked to about the paper this morning seemed confused about this, with at least one person assuming J-space was a subspace of the residual stream. So I figured this was worth clarifying.
I think this post is criminally underrated. Over half a year later, I still regularly return to this construction for inspiration when thinking about computation in superposition.
In that case, sounds kind of tough to me? If it’s constant fixed steering, you can’t have it be on when there’s German in the input and off when there isn’t any. So you might be messing up the activations a little bit on every forward pass with your negative German steering.
I haven’t actually done any activation steering though.
In my mind is the question of “how should I update on VPD/paramerer decomposition”: Is this result surprising, or would the same have happened with something like an SAE? (An SAE has the same advantage of having seen lots of German / languages tokens, and having done the autointerp work.) Would it be easy for you to test this?
Not that easy. For starters, I’m unsure what the closest SAE edit would even look like here. SAE directions don’t live in weight space. Do we make a LoRA with its right or left singular vectors hard-wired to be parallel to some SAE decoder direction? In that case the other singular vector would still be completely free to vary, so it would be a pretty different kind of fine tune than the component based edit, with way more parameters.
You could give up on editing the model itself and just make a steering vector I guess?
- ^
More generally, you can’t just ‘fine-tune in the SAE basis’, because the SAE isn’t a basis for anything, it’s overcomplete. If you try to write the model weight gradient on the backwatd pass in the ‘basis’ of the SAE decoder elements, you can’t. You can insert the SAE on the forward pass and backprop through it on the backward pass, but the resulting gradients probably end up very different from those in the target model, because SAEs are trained to reconstruct activations on the forward pass, not gradients on the backward pass. You could try to make a new kind of SAE that’s trained to reconstruct gradients on the backward pass as well as activations on the forward pass I guess? IIRC some people may have tried something like this at some point. Unsure whether they got it working though.
- ^
The parameter decomposition repo is all public. The code I used for the fine tuning isn’t, but it was really basic stuff an agent hacked together.
the CoT has changed much more between 2022 and 2026 than the forward pass has (and this wouldn’t have been obvious ex-ante)
I remember thinking in 2022 that this was obviously going to happen.
If anything the difference has been less extreme so far than past me guessed.
In a sense, it is extremely natural and obvious that any system handling sophisticated problems will be doing different things when handling different problems! But there is also a starting point from which this can be somewhat surprising: if you think of a neural net as a circuit (either just manifestly, or under some translation), then maybe you’d expect the same variables to be computed on each forward pass? It could be helpful here to consider how a Turing machine with a runtime bound can always be unrolled into a circuit that simulates [the contents of its tape and the position of its pointer] at all time steps.6 Whether tape cell 13 has a 0 or a 1 written on it at time step 42 is in one sense the same variable on any input, but in another sense it can easily represent very different variables of the program on different inputs.
Two remarks on how the current (March 2026) field trying to understand what AIs are doing relates to the issue of an AI doing different things on different inputs:
The currently prevailing view in interpretability allows for this to some extent: it is common to think of a big transformer language model as doing various different things depending on the context/input. But the prevailing view still takes there to ultimately be some pre-determined finite list of variables (I mean: corresponding to SAE features) that could be getting determined in a model, and I think this is probably a defect of that view, because a system solving an open-ended variety of complicated problems should be able to determine [what auxiliary problems to solve]/[what auxiliary questions to answer] on the fly.7 (I should note: maybe it is not clear that a forward pass of a transformer is sophisticated enough for this to be true of it?)
As one of the “finite list of variables” people[1]: This is because at the moment, I primarily want to find and understand the variables underlying the general mechanisms which AIs use to have many different kinds of productive thoughts in the first place. I am not particularly trying to find and understand variables defined only within the causal structure of these thoughts. I believe the former might indeed be described as a pre-determined finite list of variables. I agree the latter can’t be, at least not usefully.[2]
To use your analogy: I think of myself as trying to understand something like the basic makeup of a UTM, figuring out the tapes, heads, registers, tables and so on. I am not yet trying to say very much about the inner structures of the many different programs that could be run on that UTM.
I agree that some “finite list of variables” people seem to me to not distinguish between these different levels. I think that this is probably a mistake.
Could you explain that in more detail? I am looking at full example texts and failing to see a pattern like this. What is the relationship between the firing magnitude and the distance from the average position of previous occurrences?