Thanks! Broadly agreed
For example, I think our understanding of Grokking in late 2022 turned out to be importantly incomplete.
I’d be curious to hear more about what you meant by this
Thanks! Broadly agreed
For example, I think our understanding of Grokking in late 2022 turned out to be importantly incomplete.
I’d be curious to hear more about what you meant by this
I agree pretty strongly with Neel’s first point here, and I want to expand on it a bit: one of the biggest issues with interp is fooling yourself and thinking you’ve discovered something profound when in reality you’ve misinterpreted the evidence. Sure, you’ve “understood grokking”[1] or “found induction heads”, but why should anyone think that you’ve done something “real”, let alone something that will help with future dangerous AI systems? Getting rigorous results in deep learning in general is hard, and it seems empirically even harder in (mech) interp.
You can try to get around this by being extra rigorous and building from the ground up anyways. If you can present a ton of compelling evidence at every stage of resolution for your explanation, which in turn explains all of the behavior you care about (let alone a proof), then you can be pretty sure you’re not fooling yourself. (But that’s really hard, and deep learning especially has not been kind to this approach.) Or, you can try to do something hard and novel on a real system, that can’t be done with existing knowledge or techniques. If you succeed at this, then even if your specific theory is not necessarily true, you’ve at least shown that it’s real enough to produce something of value. (This is a fancy of way of saying, “new theories should make novel predictions/discoveries and test them if possible”.)
From this perspective, studying refusal in LLMs is not necessarily more x-risk relevant than studying say, studying why LLMs seem to hallucinate, why linear probes seem to be so good for many use cases(and where they break), or the effects of helpfulness/agency/tool-use finetuning in general. (And I suspect that poking hard at some of the weird results from the cyborgism crowd may be more relevant.) But it’s a hard topic that many people care about, and so succeeding here provides a better argument for the usefulness of their specific model internals based approach than studying something more niche.
It’s “easier”to study harmlessness than other comparably important or hard topics. Not only is there a lot of financial interest from companies, there’s a lot of supporting infrastructure already in place to study harmlessness. If you wanted to study the exact mechanism by which Gemini Ultra is e.g. so good at confabulating undergrad-level mathematical theorems, you’d immediately run into the problem that you don’t have Gemini internals access (and even if you do, the code is almost certainly not set up for easily poking around inside the model). But if you study a mechanism like refusal training, where there are open source models that are refusal trained and where datasets and prior work is plentiful, you’re able to leverage existing resources.
Many of the other things AI Labs are pushing hard on are just clear capability gains, which many people morally object to. For example, I’m sure many people would be very interested if mech interp could significantly improve pretraining, or suggest more efficient sparse architectures. But I suspect most x-risk focused people would not want to contribute to these topics.
Now, of course, there’s the standard reasons why it’s bad to study popular/trendy topics, including conflating your line of research with contingent properties of the topics (AI Alignment is just RLHF++, AI Safety is just harmlessness training), getting into a crowded field, being misled by prior work, etc. But I’m a fan of model internals researchers (esp mech interp researchers) apply their research to problems like harmlessness, even if it’s just to highlight the way in which mech interp is currently inadequate for these applications.
Also, I would be upset if people started going “the reason this work is x-risk relevant is because of preventing jailbreaks” unless they actually believed this, but this is more of a general distaste for dishonesty as opposed to jailbreaks or harmlessness training in general.
(Also, harmlessness training may be important under some catastrophic misuse scenarios, though I struggle to imagine a concrete case where end user-side jailbreak-style catastrophic misuse causes x-risk in practice, before we get more direct x-risk scenarios from e.g. people just finetuning their AIs to in dangerous ways.)
For example, I think our understanding of Grokking in late 2022 turned out to be importantly incomplete.
It was added recently and just added to a new release, so pip install transformer_lens
should work now/soon (you want v1.16.0 I think), otherwise you can install from the Github repo
The “love minus hate” thing really holds up
I think this discussion is sad, since it seems both sides assume bad faith from the other side. On one hand, I think Dan H and Andy Zou have improved the post by suggesting writing about related work, and signal-boosting the bypassing refusal result, so should be acknowledged in the post (IMO) rather than downvoted for some reason. I think that credit assignment was originally done poorly here (see e.g. “Citing others” from this Chris Olah blog post), but the authors resolved this when pushed.
But on the other hand, “Section 6.2 of the RepE paper shows exactly this” and accusations of plagiarism seem wrong @Dan H. Changing experimental setups and scaling them to larger models is valuable original work.
(Disclosure: I know all authors of the post, but wasn’t involved in this project)
(ETA: I added the word “bypassing”. Typo.)
transformer_lens doesn’t seem to be updated for Llama 3? Was trying to replicate Llama 3 results, would be grateful for any pointers. Thanks
We do weight editing in the RepE paper (that’s why it’s called RepE instead of ActE)
I looked at the paper again and couldn’t find anywhere where you do the type of weight-editing this post describes (extracting a representation and then changing the weights without optimization such that they cannot write to that direction).
The LoRRA approach mentioned in RepE finetunes the model to change representations which is different.
There’s been a fair amount of work on activation steering and similar techniques,, with bearing in eg sycophancy and truthfulness, where you find the vector and inject it eg Rimsky et al and Zou et al. It seems to work decently well. We found it hard to bypass refusal by steering and instead got it to work by ablation, which I haven’t seen much elsewhere, but I could easily be missing references
I agree you investigate a bunch of the stuff I mentioned generally somewhere in the paper, but did you do this for refusal-removal in particular? I spent some time on this problem before and noticed that full refusal ablation is hard unless you get the technique/vector right, even though it’s easy to reduce refusal or add in a bunch of extra refusal. That’s why investigating all the technique parameters in the context of refusal in particular is valuable.
is novel compared to… RepE
This is inaccurate, and I suggest reading our paper: https://arxiv.org/abs/2310.01405
Demonstrate full ablation of the refusal behavior with much less effect on coherence
In our paper and notebook we show the models are coherent.
Investigate projection
We did investigate projection too (we use it for concept removal in the RepE paper) but didn’t find a substantial benefit for jailbreaking.
harmful/harmless instructions
We use harmful/harmless instructions.
Find that projecting away the (same, linear) feature at all layers improves upon steering at a single layer
In the RepE paper we target multiple layers as well.
Test on many different models
The paper used Vicuna, the notebook used Llama 2. Throughout the paper we showed the general approach worked on many different models.
Describe a way of turning this into a weight-edit
We do weight editing in the RepE paper (that’s why it’s called RepE instead of ActE).
but generally people should be free to post research updates on LW/AF that don’t have a complete thorough lit review / related work section.
I agree if they simultaneously agree that they don’t expect the post to be cited. These can’t posture themselves as academic artifacts (“Citing this work” indicates that’s the expectation) and fail to mention related work. I don’t think you should expect people to treat it as related work if you don’t cover related work yourself.
Otherwise there’s a race to the bottom and it makes sense to post daily research notes and flag plant that way. This increases pressure on researchers further.
including refusal-bypassing-related ones
The prior work that is covered in the document is generally less related (fine-tuning removal of safeguards, truth directions) compared to these directly relevant ones. This is an unusual citation pattern and gives the impression that the artifact is making more progress/advancing understanding than it actually is.
I’ll note pretty much every time I mention something isn’t following academic standards on LW I get ganged up on and I find it pretty weird. I’ve reviewed, organized, and can be senior area chair at ML conferences and know the standards well. Perhaps this response is consistent because it feels like an outside community imposing things on LW.
FWIW I published this Alignment Forum post on activation steering to bypass refusal (albeit an early variant that reduces coherence too much to be useful) which from what I can tell is the earliest work on linear residual-stream perturbations to modulate refusal in RLHF LLMs.
I think this post is novel compared to both my work and RepE because they:
Demonstrate full ablation of the refusal behavior with much less effect on coherence / other capabilities compared to normal steering
Investigate projection thoroughly as an alternative to sweeping over vector magnitudes (rather than just stating that this is possible)
Find that using harmful/harmless instructions (rather than harmful vs. harmless/refusal responses) to generate a contrast vector is the most effective (whereas other works try one or the other), and also investigate which token position at which to extract the representation
Find that projecting away the (same, linear) feature at all layers improves upon steering at a single layer, which is different from standard activation steering
Test on many different models
Describe a way of turning this into a weight-edit
Edit:
(Want to flag that I strong-disagree-voted with your comment, and am not in the research group—it is not them “dogpiling”)
I do agree that RepE should be included in a “related work” section of a paper but generally people should be free to post research updates on LW/AF that don’t have a complete thorough lit review / related work section. There are really very many activation-steering-esque papers/blogposts now, including refusal-bypassing-related ones, that all came out around the same time.
From Andy Zou:
Thank you for your reply.
Model interventions to bypass refusal are not discussed in Section 6.2.
We perform model interventions to robustify refusal (your section on “Adding in the “refusal direction” to induce refusal”). Bypassing refusal, which we do in the GitHub demo, is merely adding a negative sign to the direction. Either of these experiments show refusal can be mediated by a single direction, in keeping with the title of this post.
we examined Section 6.2 carefully before writing our work
Not mentioning it anywhere in your work is highly unusual given its extreme similarity. Knowingly not citing probably the most related experiments is generally considered plagiarism or citation misconduct, though this is a blog post so norms for thoroughness are weaker. (lightly edited by Dan for clarity)
Ablating vs. Addition
We perform a linear combination operation on the representation. Projecting out the direction is one instantiation of it with a particular coefficient, which is not necessary as shown by our GitHub demo. (Dan: we experimented with projection in the RepE paper and didn’t find it was worth the complication. We look forward to any results suggesting a strong improvement.)
--
Please reach out to Andy if you want to talk more about this.
Edit: The work is prior art (it’s been over six months+standard accessible format), the PIs are aware of the work (the PI of this work has spoken about it with Dan months ago, and the lead author spoke with Andy about the paper months ago), and its relative similarity is probably higher than any other artifact. When this is on arXiv we’re asking you to cite the related work and acknowledge its similarities rather than acting like these have little to do with each other/not mentioning it. Retaliating by some people dogpile voting/ganging up on this comment to bury sloppy behavior/an embarrassing oversight is not the right response (went to −18 very quickly).
Edit 2: On X, Neel “agree[s] it’s highly relevant” and that he’ll cite it. Assuming it’s covered fairly and reasonably, this resolves the situation.
Edit 3: I think not citing it isn’t a big deal because I think of LW as a place for ml research rough drafts, in which errors will happen. But if some are thinking it’s at the level of an academic artifact/is citable content/is an expectation others cite it going forward, then failing to mention extremely similar results would actually be a bigger deal. Currently I’ll think it’s the former.
This is great work, but I’m a bit disappointed that x-risk-motivated researchers seem to be taking the “safety”/”harm” framing of refusals seriously. Instruction-tuned LLMs doing what their users ask is not unaligned behavior! (Or at best, it’s unaligned with corporate censorship policies, as distinct from being unaligned with the user.) Presumably the x-risk-relevance of robust refusals is that having the technical ability to align LLMs to corporate censorship policies and against users is better than not even being able to do that. (The fact that instruction-tuning turned out to generalize better than “safety”-tuning isn’t something anyone chose, which is bad, because we want humans to actively choosing AI properties as much as possible, rather than being at the mercy of which behaviors happen to be easy to train.) Right?
We definitely drew inspiration from the Representation Engineering paper and other activation steering papers, but we think our work is quite distinct.
In particular, we examined Section 6.2 carefully before writing our work, and we do not see it showing the same result that we show here.
Here’s my summary of Section 6.2:
Section 6.2.1 obtains reading vectors using contrastive pairs of harmful and harmless instructions, and then uses these reading vectors for 90% classification accuracy between harmful and harmless instructions. The authors then append jailbreaks to the prompts, which cause the model not to refuse, and observe that the reading vectors still obtain 90% classification accuracy on distinguishing harmful vs harmless instructions. This means that the reading vectors are not representing refusal, but rather they are representing whether the instruction is harmful or harmless. In fact, the point of this experiment is to show that these are distinct.
To quote the conclusion of Section 6.2.1: “This compelling evidence suggests the presence of a consistent internal concept of harmfulness that remains robust to such perturbations, while other factors must account for the model’s choice to follow harmful instructions, rather than perceiving them as harmless.”
Section 6.2.2 describes an intervention to improve model robustness to jailbreaks, i.e. to increase the rate of refusals on harmful instructions when jailbreaks are appended to them. They do this by amplifying the harmfulness feature whenever it is detected, which obtains a higher refusal rate.
Section 6.2 only considers a single model, Vicuna-13B.
We would agree that using established techniques from representation engineering / activation steering to induce refusal is not novel. Inducing refusal via activation addition is quite easy in our experience.
However, the main result of our work is that we found an intervention that bypasses refusal consistently while also maintaining model coherence. Model interventions to bypass refusal are not discussed in Section 6.2.
As for the demo notebook in the representation-engineering repo—we were not previously aware of this notebook. The result of bypassing refusal is not reported in the paper, and so we didn’t think to look through the repo.
That being said, the notebook shows an intervention for a single prompt on a single model. Anecdotally, we tried doing vanilla activation addition with the negative “refusal direction” at particular layers, and we were not able to consistently bypass refusal while also maintaining model coherence. If there is a methodology involving activation addition (rather than ablation, as we did here), we would be interested in seeing a more thorough demonstration across prompts and models. We’d also be interested in comparing the two methodologies across metrics measuring refusal and coherence.
I’d also be happy to hop on a call if you’d like to discuss further.
From Andy Zou:
Section 6.2 of the Representation Engineering paper shows exactly this (video). There is also a demo here in the paper’s repository which shows that adding a “harmlessness” direction to a model’s representation can effectively jailbreak the model.
Going further, we show that using a piece-wise linear operator can further boost model robustness to jailbreaks while limiting exaggerated refusal. This should be cited.
Sorry for maybe naive question. Which other behaviors X could be defeated by this technique of “find n instructions that induce X and n that don’t”? Would it work for X=unfriendliness, X=hallucination, X=wrong math answers, X=math answers that are wrong in one specific way, and so on?
Thanks!
I was grouping that with “the computation may require mixing together ‘natural’ concepts” in my head. After all, entropy isn’t an observable in the environment, it’s something you derive to better model the environment. But I agree that “the concept may not be one you understand” seems more central.
Thank you, I’ve been hoping someone would write this disclaimer post.
I’d add on another possible explanation for polysemanticity, which is that the model might be thinking in a limited number of linearly represented concepts, but those concepts need not match onto concepts humans are already familiar with. At least not all of them.
Just because the simple meaning of a direction doesn’t jump out at an interp researcher when they look at a couple of activating dataset examples doesn’t mean it doesn’t have one. Humans probably wouldn’t even always recognise the concepts other humans think in on sight.
Imagine a researcher who hasn’t studied thermodynamics much looking at a direction in a model that tracks the estimated entropy of a thermodynamic system it’s monitoring: ‘It seems to sort of activate more when the system is warmer. But that’s not all it’s doing. Sometimes it also goes up when two separated pockets of different gases mix together, for example. Must be polysemantic.’
Yeah, sorry that was unclear; there’s no need for any form of hypercomputation to get an enumeration of the axioms of U. But you need a halting oracle to distinguish between the axioms and non-axioms. If you don’t care about distinguishing axioms from non-axioms, but you do want to get an assignment of truthvalues to the atomic formulas Q(i,j) that’s consistent with the axioms of U, then that is applying a consistent guessing oracle to U.