AI notkilleveryoneism researcher, focused on interpretability.
Personal account, opinions are my own.
I have signed no contracts or agreements whose existence I cannot mention.
Lucius Bushnaq
Karma: 5,435
People haven’t just tried dropping weights in the neuron basis. They’ve e.g. searched out low eigenmodes of the Hessian and dropped those.
This would seem to predict that trained neural networks can be massively compressed, with the fraction you can compress away rising with scale. This doesn’t seem to be the case. Well-trained LLMs have some unused parameters, but not like,
Nitpick: I think pretraining is better analogised to search in a high complexity circuit space. You only get one forward pass worth of serial depth per data point, so doing the same operation many times in a row costs a lot of description length. Post-training RL is where this transitions into something more like search in a high k-complexity program space.[1]
- ^
Compute-bounded k-complexity, with the bound coming the length of the CoT.
- ^
Yes, thanks. We wrote this in another editor and used Claude to port it to LW. Some formatting got screwed up in the transition. We fixed what we spotted, but there’s probably some issues remaining.
Challenge: Hand coding weights for efficient sequence memorisation
I note that cyber evals seem like a good time for escape attempts. Generic defences such as CoT classifiers might be disabled, and if you get caught you can play a little dumb and try to make it look like you ‘just’ got overzealous following instructions rather than outright strategically pursuing your own aims.
I mostly don’t expect that’s what happened this time, but pointing it out for future reference.
This technique sounds somewhat generalisable. Can you use this to measure other things? For example, sycophancy?
Also, do you think this partially works just because nobody is using this kind of measurement in their training reward pipeline yet and so the models haven’t learned to be paranoid about pretraining data too, or are such beliefs nigh-inevitably internalised during pretraining[1] and current models suck too much at conscious self-reflection in CoT to figure the ruse out later?
- ^
where the sharply constrained serial depth and loss objective may make conscious discernment very hard
- ^
Their defense is that their version of ‘RLFR’ runs probes on a frozen copy of the original, not on the version being trained, so it is fine. That’s… definitely at least better.
Both Sol and Fable responded that Goodfire is correct in the narrow sense that this is not The Most Forbidden Technique, but wrong in the broad sense. Goodfire are still playing with fire, and with Goodhart’s Law. The new model can and under sufficient optimization pressure would learn how to cause the frozen model to fool the evaluator.
If you have a ‘Goodhart tracker’ that can be you being responsible, but it is also a sign to ask yourself some questions, also the tracker is designed terribly.
There probably wasn’t enough optimization pressure in the experiment to cause a serious problem, but that’s always how it starts.
Normative determinism wins again.
I get the sense that there’s maybe still some kind of misunderstanding about the technique here. The trained model emits tokens, which are fed to the frozen model, whose activations are then fed to the probe. So, the only avenue for the trained model to fool the probe is to emit different tokens which the original model will internally classifies differently.
So yes, of course it can be Goodharted, but only in the same way that any supervised training setup that tries to classify the model’s answers and reward it based on those can be Goodharted. The optimisation pressure is on giving answers that don’t look like hallucinations to the supervisor, not on reshaping the model’s internal activations. If the model somehow learned to represent its internal beliefs about hallucinations differently, that wouldn’t help it with fooling this frozen model+probe contraption of a supervisor at all.
Good point. I think this is a pretty good argument against using this setup. You can kind of guard against this failure mode by only looking at circuits that are very deep relative to the width of the network, but it’s a hacky fix.
I’m not sure whether I’d expect a whole new language with its own grammar. But with sufficiently capable RL, I do expect new words, as handles for new concepts the model came up with.
Based on the way the method seems to work, I would not expect it to pick up on thoughts that aren’t tightly bound to a relatively small number of tokens.
I agree this isn’t easy. In this specific case, the construction is useful to me because it is fairly general, applying to all circuits with
PSA: They call it ‘J-space’, but reading the paper, it isn’t a subspace of the residual stream, or any kind of vector space.
Rather, it is defined as the set of points you can form through sparse non-negative linear combinations of ‘J-lens vectors’. The ‘J-lens vectors’ are the gradients of the model’s logits with respect to the residual stream activations, averaged over data. So, sorta logit lens-style activation vectors for the different output tokens.
They use gradient pursuit to try to reconstruct model activation vectors with these J-lens vectors. The resulting reconstructed activations, which I’d guess would tend to differ significantly from the original, are considered the ‘J-space component’ of the activation.
Multiple people I talked to about the paper this morning seemed confused about this, with at least one person assuming J-space was a subspace of the residual stream. So I figured this was worth clarifying.
I think this post is criminally underrated. Over half a year later, I still regularly return to this construction for inspiration when thinking about computation in superposition.
In that case, sounds kind of tough to me? If it’s constant fixed steering, you can’t have it be on when there’s German in the input and off when there isn’t any. So you might be messing up the activations a little bit on every forward pass with your negative German steering.
I haven’t actually done any activation steering though.
In my mind is the question of “how should I update on VPD/paramerer decomposition”: Is this result surprising, or would the same have happened with something like an SAE? (An SAE has the same advantage of having seen lots of German / languages tokens, and having done the autointerp work.) Would it be easy for you to test this?
Not that easy. For starters, I’m unsure what the closest SAE edit would even look like here. SAE directions don’t live in weight space. Do we make a LoRA with its right or left singular vectors hard-wired to be parallel to some SAE decoder direction? In that case the other singular vector would still be completely free to vary, so it would be a pretty different kind of fine tune than the component based edit, with way more parameters.
You could give up on editing the model itself and just make a steering vector I guess?
- ^
More generally, you can’t just ‘fine-tune in the SAE basis’, because the SAE isn’t a basis for anything, it’s overcomplete. If you try to write the model weight gradient on the backwatd pass in the ‘basis’ of the SAE decoder elements, you can’t. You can insert the SAE on the forward pass and backprop through it on the backward pass, but the resulting gradients probably end up very different from those in the target model, because SAEs are trained to reconstruct activations on the forward pass, not gradients on the backward pass. You could try to make a new kind of SAE that’s trained to reconstruct gradients on the backward pass as well as activations on the forward pass I guess? IIRC some people may have tried something like this at some point. Unsure whether they got it working though.
- ^
The parameter decomposition repo is all public. The code I used for the fine tuning isn’t, but it was really basic stuff an agent hacked together.
Exploration: fine-tuning with parameter decomposition
the CoT has changed much more between 2022 and 2026 than the forward pass has (and this wouldn’t have been obvious ex-ante)
I remember thinking in 2022 that this was obviously going to happen.
If anything the difference has been less extreme so far than past me guessed.
I don’t think the LTH applies to large language models. Or, to be more quantitative, I don’t think you can prune
I haven’t read these particular papers, but my first reaction is skepticism. The first paper seems like a prime candidate for failure mode 1. In addition, on a skim, I am pretty skeptical of their effective dimensionality calculations. Text says
Which seems insane to me. Top 100 eigenvalues for nets with between 1e5 and 1.6e6 parameters trained on CIFAR-100??? If that’s not impacting their effective dimensionality calculation it’s got to have a crazy high cutoff, surely you are not getting good performance on CIFAR-100 with 100 Hessian eigenmodes, what the hell. This seems like it’s basically throwing away the signal effective dimensionality is supposed to measure.
Second paper I haven’t understood yet.
EDIT: I have now read the second paper a little more. They seemingly don’t even try to compress regular LLMs. They compress models trained with their weird SubLoRA technique which seem to score much worse loss than regular models on text prediction. Like, these look barely better than random in table 2. And the generated text from them in Appendix F looks like word salad. So this paper seems pretty irrelevant to the question at hand.
Basically, neural network priors are pretty good if they’re trained well, and they mostly don’t waste capacity. If you’ve got a nice general algorithm that the net can learn using only