AI notkilleveryoneism researcher, focused on interpretability.
Personal account, opinions are my own.
I have signed no contracts or agreements whose existence I cannot mention.
Lucius Bushnaq
Karma: 5,450
Nice, thanks!
Also for the challenge proposed, is there a particular area to post ideas?You can just drop a link here in the comments.
A. To store new facts
B. To remove facts cleanly (unlearning)
C. To find facts (ie we have a bilinear layer that’s half-facts and half-structure, and we want to find the fact-half?)Ideally all three, but I especially care about the third one. Generally, we want to not be as confused when we stare at language model weights. Knowing something about the way facts tend to be encoded in weights might make us less confused about at least some of the weights’ structure. Maybe a lot of the structure, if a lot of the weights’ content by bit count is some form of memorised sequence data.
There’s a closed form solution for bilinear layers, but it’s basically SGD (it relates to the fixed points found by KKT, though I can include a better explanation later). This efficient solution basically relies on the facts being assumed by other facts in a connected web of facts (where adding a new fact loosens the web, then pulls all other facts taut, so you have to update everything learned so far).
Nice! I figured there might be some kind of closed form solution for bilinear layers, but I didn’t know what it was. For a while we were playing with the idea of searching for such a solution and then trying to transfer it to the ReLU context, but we ended up parking that.
Dan Braun and me have had a lot of success in the past few days getting agents to grind on the challenge, now that this post exists as a prompt. They’ve already found hand-coded solutions that IIRC seem to perform basically as well as the trained solutions, sometimes a little better. I’ve now switched to having them hunt for solutions that also look like the trained solution if you just stare at the weights and neuron patterns. I figured that objective might be too fuzzy for them, but it seems like they might be making progress on that as well.
Waiting to post any of that until I’ve verified it a bit more though.
Tokens which occurred earlier in context window. With distance from the average position of previous occurrences determining the strength (so tends to fire on when a token was repeated a while back).
Could you explain that in more detail? I am looking at full example texts and failing to see a pattern like this. What is the relationship between the firing magnitude and the distance from the average position of previous occurrences?
The LTH gives us a way to do this! It’s just so computationally expensive (iterative neuron pruning and evaluation) as to be useless.
I don’t think the LTH applies to large language models. Or, to be more quantitative, I don’t think you can prune
It actually is! See here, which kind of gets at the question by finding that effective dimensionality of CIFAR NNs decreases with scale, and here, which directly identifies that “larger models have better generalization bounds and are more compressible than smaller models.”
I haven’t read these particular papers, but my first reaction is skepticism. The first paper seems like a prime candidate for failure mode 1. In addition, on a skim, I am pretty skeptical of their effective dimensionality calculations. Text says
Lanczos Calculations We use GPU enabled Lanczos as implemented in Gardner et al. (2018) to compute the eigenvalues approximately, running 100 steps to compute approximately 100 of the top eigenvalues. We note that our estimates of the effective dimensionality are somewhat biased from not including all of the small eigenvalues. However, these small eigenvalues will contribute negligibly and Lanczos will converge to the true eigenvalues if we ran k steps where k is the rank of the Hessian.
Which seems insane to me. Top 100 eigenvalues for nets with between 1e5 and 1.6e6 parameters trained on CIFAR-100??? If that’s not impacting their effective dimensionality calculation it’s got to have a crazy high cutoff, surely you are not getting good performance on CIFAR-100 with 100 Hessian eigenmodes, what the hell. This seems like it’s basically throwing away the signal effective dimensionality is supposed to measure.
Second paper I haven’t understood yet.
EDIT: I have now read the second paper a little more. They seemingly don’t even try to compress regular LLMs. They compress models trained with their weird SubLoRA technique which seem to score much worse loss than regular models on text prediction. Like, these look barely better than random in table 2. And the generated text from them in Appendix F looks like word salad. So this paper seems pretty irrelevant to the question at hand.Basically, neural network priors are pretty good if they’re trained well, and they mostly don’t waste capacity. If you’ve got a nice general algorithm that the net can learn using only
People haven’t just tried dropping weights in the neuron basis. They’ve e.g. searched out low eigenmodes of the Hessian and dropped those.
This would seem to predict that trained neural networks can be massively compressed, with the fraction you can compress away rising with scale. This doesn’t seem to be the case. Well-trained LLMs have some unused parameters, but not like,
Nitpick: I think pretraining is better analogised to search in a high complexity circuit space. You only get one forward pass worth of serial depth per data point, so doing the same operation many times in a row costs a lot of description length. Post-training RL is where this transitions into something more like search in a high k-complexity program space.[1]
- ^
Compute-bounded k-complexity, with the bound coming the length of the CoT.
- ^
Yes, thanks. We wrote this in another editor and used Claude to port it to LW. Some formatting got screwed up in the transition. We fixed what we spotted, but there’s probably some issues remaining.
Challenge: Hand coding weights for efficient sequence memorisation
I note that cyber evals seem like a good time for escape attempts. Generic defences such as CoT classifiers might be disabled, and if you get caught you can play a little dumb and try to make it look like you ‘just’ got overzealous following instructions rather than outright strategically pursuing your own aims.
I mostly don’t expect that’s what happened this time, but pointing it out for future reference.
This technique sounds somewhat generalisable. Can you use this to measure other things? For example, sycophancy?
Also, do you think this partially works just because nobody is using this kind of measurement in their training reward pipeline yet and so the models haven’t learned to be paranoid about pretraining data too, or are such beliefs nigh-inevitably internalised during pretraining[1] and current models suck too much at conscious self-reflection in CoT to figure the ruse out later?
- ^
where the sharply constrained serial depth and loss objective may make conscious discernment very hard
- ^
Their defense is that their version of ‘RLFR’ runs probes on a frozen copy of the original, not on the version being trained, so it is fine. That’s… definitely at least better.
Both Sol and Fable responded that Goodfire is correct in the narrow sense that this is not The Most Forbidden Technique, but wrong in the broad sense. Goodfire are still playing with fire, and with Goodhart’s Law. The new model can and under sufficient optimization pressure would learn how to cause the frozen model to fool the evaluator.
If you have a ‘Goodhart tracker’ that can be you being responsible, but it is also a sign to ask yourself some questions, also the tracker is designed terribly.
There probably wasn’t enough optimization pressure in the experiment to cause a serious problem, but that’s always how it starts.
Normative determinism wins again.
I get the sense that there’s maybe still some kind of misunderstanding about the technique here. The trained model emits tokens, which are fed to the frozen model, whose activations are then fed to the probe. So, the only avenue for the trained model to fool the probe is to emit different tokens which the original model will internally classifies differently.
So yes, of course it can be Goodharted, but only in the same way that any supervised training setup that tries to classify the model’s answers and reward it based on those can be Goodharted. The optimisation pressure is on giving answers that don’t look like hallucinations to the supervisor, not on reshaping the model’s internal activations. If the model somehow learned to represent its internal beliefs about hallucinations differently, that wouldn’t help it with fooling this frozen model+probe contraption of a supervisor at all.
Good point. I think this is a pretty good argument against using this setup. You can kind of guard against this failure mode by only looking at circuits that are very deep relative to the width of the network, but it’s a hacky fix.
I’m not sure whether I’d expect a whole new language with its own grammar. But with sufficiently capable RL, I do expect new words, as handles for new concepts the model came up with.
Based on the way the method seems to work, I would not expect it to pick up on thoughts that aren’t tightly bound to a relatively small number of tokens.
I agree this isn’t easy. In this specific case, the construction is useful to me because it is fairly general, applying to all circuits with
PSA: They call it ‘J-space’, but reading the paper, it isn’t a subspace of the residual stream, or any kind of vector space.
Rather, it is defined as the set of points you can form through sparse non-negative linear combinations of ‘J-lens vectors’. The ‘J-lens vectors’ are the gradients of the model’s logits with respect to the residual stream activations, averaged over data. So, sorta logit lens-style activation vectors for the different output tokens.
They use gradient pursuit to try to reconstruct model activation vectors with these J-lens vectors. The resulting reconstructed activations, which I’d guess would tend to differ significantly from the original, are considered the ‘J-space component’ of the activation.
Multiple people I talked to about the paper this morning seemed confused about this, with at least one person assuming J-space was a subspace of the residual stream. So I figured this was worth clarifying.
I think this post is criminally underrated. Over half a year later, I still regularly return to this construction for inspiration when thinking about computation in superposition.
In that case, sounds kind of tough to me? If it’s constant fixed steering, you can’t have it be on when there’s German in the input and off when there isn’t any. So you might be messing up the activations a little bit on every forward pass with your negative German steering.
I haven’t actually done any activation steering though.
I’m not aware of Redwood or METR having security expertise. If they indeed do not, I think it would be better if they partnered with a security company for this.