protest about what exactly?
jacob_drori
Karma: 818
LHT says (emphasis mine):
dense, randomly-initialized, feed-forward networks contain subnetworks (“winning tickets”) that—when trained in isolation—reach test accuracy comparable to the original network in a similar number of iterations
Note that in LHT the subnetworks are trained, unlike in the results I cited.
Weight-Sparse Circuits May Be Interpretable Yet Unfaithful
Not really. What properties does an “interpretability” system have that are not shared any old system in the brain?
My guess is doing it with RL is worse because I expect it to generalize to more probes
Why?
I got lost at the second diagram. Why is the process of computing expected reward for a given thought/plan called “interpretability”? In what way is it analogous to mech interp? I’d find it clarifying to see a sketch of an interpretability-in-the-loop ML training setup with the same rough arrows as in that diagram.
I won’t argue about definitions of faithfulness—yours seems as fine as any—but I think I only care about unfaithfulness in as far as it causes our circuits to make incorrect predictions about OOD behavior.
For the toy model, I’d like it if you explicitly showed that there are inputs on which the CLT replacement model gives very different outputs to the original model. You could then check whether the PLT replacement model performs better than the CLT one on these inputs. This result should be simple to obtain: just choose a nonbinary input, i.e. an that is not in . But it would make the toy model more compelling to me, since, to me, faithfulness is really about OOD generalization.
[Question] Does the USG have access to smarter models than the labs’?
do you extract the concept vector at the same layer you steer at?
an additional frequency penalty which prevented the formation of high-frequency features
Please could you write the equation for this loss term? I couldn’t find it in the paper.
Is the “CoT Monitor Detection Rate” the fraction of CoTs in which the fully expanded derivative (i.e. the “bad behavior”) is present?
Yes.
If so, doesn’t this show that the Mind & Face method is working (very slight improvement over the Penalty method)? The task reward is high, the output penalty is low, and the rate of bad behavior (expanded derivative in CoT) is also low. How can there be spillover if the bad behavior simply doesn’t occur at all?
Well noticed. Yeah, I admit this is confusing. Under the natural interpretation of “bad behavior” as meaning “writing the expanded derivative anywhere”, it’s true that the bad behavior was successfully removed. Therefore we interpret the polynomial environment as an example where training against an output monitor caused a big change to the sort of reasoning being done in the CoT. As you note, it is not an example of obfuscation.
In our ICLR revision, we clarified this a bit. I think the fact that “obfuscated” appears in the paper’s title may lead to others having the same, very reasonable confusion as you did, which is a shame but I’m not sure what to do about it.(There’s an alternate, less natural interpretation of “bad behavior” as meaning “factoring the derivative at all”. Under this interpretation, the polynomial environment gives an example of obfuscation, since the model continues to factor the polynomial, just in a way our monitor can’t detect. However, this interpretation of “bad behavior” is weird, since it’s precisely the thing we’re training for! That is, it’s weird to reward the model for factoring the derivative but then turn around and call it “bad behavior”)
[Paper] Output Supervision Can Obfuscate the CoT
Thanks for the very clear reply. I like your decomposition of CAFT into (1) ablating activations and (2) ablating gradients.
To isolate effect (1), we can test a variant of CAFT where we detach the projection before we subtract it from the activations, such that the gradients through the subspace are not ablated.
...
We did this experiment on the two models used in the paper: for one of them, this version performs slightly worse but similarly to CAFT and, for the other one, it doesn’t reduce misalignment at all and it’s similar to regular finetuning.I predict that for the model where only-ablate-activations performed well, the ablation is actually steering towards misalignment (whereas for the other model, zero-ablation is steering away from, or orthogonally to, misalignment). I’ll check this at some point in the next week or so.
I think the general technique of train-time steering seems promising enough that it’s worth figuring out best practices. These best practices might depend on the task:
If the base model already contains circuitry that reads from the steering direction and produces the bad behavior (as is the case with EM) then preventative steering seems to make sense.
But if the base model does not already have that circuitry, perhaps we should just mean-ablate the direction, making it harder for the bad circuitry to be learnt in the first place.
Ofc, this is pure speculation. I just want to highlight the fact that there seems to be a lot still to be understood.
To begin talking about volume, you first need to really understand what space is.
No, stop it, this is a terrible approach to math education. “Ok kids, today we’re learning about the area of a circle. First, recall the definition of a manifold.” No!!
Persona Vectors notes a confusing aspect of CAFT (see appendix J.1):
We hypothesize that CAFT’s effectiveness in the evil and sycophancy cases stems from the fact that forcing activations to have zero projection effectively acts as positive preventative steering (because of the initial negative projections in the base model).
I agree with them. Zero-ablating a subspace is unprincipled, since the residual stream has no preferred origin. The practical consequence is that for behaviors, like EM, that can be triggered by adding a constant steering vector , CAFT’s zero-ablation may actually increase the component along and thus steer towards the behavior.
I think (though I may have missed something) that the CAFT paper gives no evidence for whether zero-ablation steers towards or away from EM, so it is hard to compare its results with those of Persona Vectors. You could resolve this confusion by:
1) Checking whether zero-ablation increases or decreases the activations of “misaligned” SAE latents
2) Checking whether zero-ablating the base model leads to more or less misaligned responses (note: no training involved).
I really thought I’d updated all the way towards “models linearly represent everything”. But somehow I was still surprised by Dmitrii’s training order result.
LLMs linearly represent the accuracy of their next-token predictions
A quick experiment I did on Llama-3.2-1B:
Choose a layer, and let be that layer’s residual activation at token position .
Let be the logit at position that was assigned to whatever turned out to be the correct next token (i.e. the correct token at position ).
Use least-squares to fit a linear map that takes and predicts .
Results for each layer are below. Layers in the second half of the model (except the last layer) have decent () linear representations of the correct logit.
I don’t find this result very surprising. I checked it because I thought it might explain what’s going on in this very cool recent paper by @Dmitrii Krasheninnikov , Richard Turner and @David Scott Krueger (formerly: capybaralet)). They finetune Llama-3.2-1B and show that the model linearly represents the order of appearance of finetuning data. If more recent data has lower loss, then perhaps my probing results explain theirs.
Ah I hadn’t noticed that, very nice. Great post!
I mean sure, but a protest should have a clear message