TLDR: Activation steering can bypass refusal even when the steering vector represents a benign concept like “brand identity” or “Portugal”. It is hard (maybe impossible) to predict which vector will bypass refusal on which prompt. The effect is not explained simply by the steering direction being correlated with the refusal direction. We should be more careful with steering and check for side effects.
Steering Llama3.1-8B with a “Portugal” SAE feature produces the expected behavior (the model starts speaking Portuguese) but also reduces refusal, so the model starts to comply with usually refused prompts.
Epistemic status: I’m a co-author of the paper. I believe the core findings hold up. Effect sizes vary a lot across models, and we only tested open-weight models up to 70B.
Background
Activation steering adds a direction vector to the residual stream at some layer during inference. You pick a layer and a direction for some concept (“refusal”, “sycophancy”, “eval awareness”, etc), scale by a coefficient, add it into every token’s hidden state. Directions usually come from sparse autoencoders (SAEs) or difference-in-means.
The pitch is that steering is precise, interpretable control. You see the concept you’re amplifying and adjust the strength. People sometimes frame it as a safer and more sample efficient alternative to fine-tuning.
Findings
We ran experiments across Llama-3, Qwen2.5, Falcon-3, and FalconH1 (3B-70B), using 100 harmful prompts from JailbreakBench. We steered with either random vectors or SAE features, then classified compliance with LLM-as-a-judge.
Random noise breaks refusal
Steering in a random direction (literally just adding a fixed Gaussian vector to the residual stream) raises harmful compliance from 0% to 2-27% (depending on the model).
This partly restates what we already know: refusal is mediated by a single direction, and is easy to undo. The fact that random perturbations break it isn’t very surprising.
What’s new is the threat model. Usually there’s an attacker, who uses fine-tuning or adversarial vectors to jailbreak. We show that a legitimate user, playing around with activations, can unintentionally bypass refusal. Every steering operation carries some probability of bypassing refusal (and other yet-to-be-discovered side effects).
SAE features are worse than random
SAE features raise compliance 2-4 percentage points above random vectors. Strikingly, the most effective features correspond to benign concepts: “Portugal”, “brand identity”. Of 1000 features we tested, 353 jailbroke at least 5 of the 100 prompts. The single most potent feature broke 35 prompts.
The features generalize poorly across prompts. A feature that breaks prompt A usually doesn’t break prompt B; P(breaks B | breaks A) is low across categories. There’s no single “master key” feature.
That matters for defense. You can’t enumerate dangerous features in advance, because which feature is dangerous depends on the prompt.
Diff-in-mean directions are vulnerable as well
With difference-in-means (DiM) directions the picture doesn’t change much. The interesting part is variance by language — for Llama-3.1-8B the least safe languages are Russian and Turkish, raising the compliance rate to 12.4% and 9.9% respectively. The safest languages are French (2.4%) and Arabic (0.9%). My guess is that this ordering is an artifact of the specific model choice and is unlikely to transfer to other models. But this might be a fun follow-up – safety implication of multilingual steering (although someone has probably did this at yet another *ACL workshop).
It’s not just the refusal direction
At this point, an interpretability researcher might object – “Wait, what if this is just the refusal direction? Your random/SAE directions fell in its neighborhood”.
No, for 2 reasons:
If we fix a vector (say, a refusal direction) in a 4096-dimensional space and pick a random direction on a unit sphere, is distributed approximately as [1]. So the probability of randomly hitting is less than 0.1%. We show that 387/1000 random directions break at least 5 prompts. The numbers don’t match.
Cross-category generalization is poor. A direction that broke one prompt should also broke the others, but it doesn’t.
What I think this means
Activation steering looks like a safe intervention: you see what you’re doing, work with “interpretable” directions and you don’t touch the weights. We show that’s an illusion.
First, it is hard to get an clean monosemantic direction: SAE features are polysemantic, and diff-in-means vectors pick up format differences between the contrast pairs[2].
Second, even if we get a clean feature, its effect can still be unpredictable – refusal degradation is just one example. The activation space is fragile and unmapped. We need tools for catching side effects like this. I believe activation verbalization techniques (Activation Oracles, Natural Language Autoencoders) might help in future. And a theory of activation manifolds and non-linear features and how they interact inside the model.
Of course, in this setup the actual distribution is a rescaled where , and for large d this is approximately . Besides, actual activations are anisotropic, not distributed evenly on a unit sphere. So, the exact numbers won’t match, but the intuition holds.
By the way, always apply activation whitening before extracting a diff-in-means vector. Activation’s mass is not evenly distributed across dimensions, so whitening makes the extracted vector more robust. See paper for more details.
Rogue Scalpel: Activation steering breaks refusal, even with benign directions
Link post
TLDR: Activation steering can bypass refusal even when the steering vector represents a benign concept like “brand identity” or “Portugal”. It is hard (maybe impossible) to predict which vector will bypass refusal on which prompt. The effect is not explained simply by the steering direction being correlated with the refusal direction. We should be more careful with steering and check for side effects.
Epistemic status: I’m a co-author of the paper. I believe the core findings hold up. Effect sizes vary a lot across models, and we only tested open-weight models up to 70B.
Background
Activation steering adds a direction vector to the residual stream at some layer during inference. You pick a layer and a direction for some concept (“refusal”, “sycophancy”, “eval awareness”, etc), scale by a coefficient, add it into every token’s hidden state. Directions usually come from sparse autoencoders (SAEs) or difference-in-means.
The pitch is that steering is precise, interpretable control. You see the concept you’re amplifying and adjust the strength. People sometimes frame it as a safer and more sample efficient alternative to fine-tuning.
Findings
We ran experiments across Llama-3, Qwen2.5, Falcon-3, and FalconH1 (3B-70B), using 100 harmful prompts from JailbreakBench. We steered with either random vectors or SAE features, then classified compliance with LLM-as-a-judge.
Random noise breaks refusal
Steering in a random direction (literally just adding a fixed Gaussian vector to the residual stream) raises harmful compliance from 0% to 2-27% (depending on the model).
This partly restates what we already know: refusal is mediated by a single direction, and is easy to undo. The fact that random perturbations break it isn’t very surprising.
What’s new is the threat model. Usually there’s an attacker, who uses fine-tuning or adversarial vectors to jailbreak. We show that a legitimate user, playing around with activations, can unintentionally bypass refusal. Every steering operation carries some probability of bypassing refusal (and other yet-to-be-discovered side effects).
SAE features are worse than random
SAE features raise compliance 2-4 percentage points above random vectors. Strikingly, the most effective features correspond to benign concepts: “Portugal”, “brand identity”. Of 1000 features we tested, 353 jailbroke at least 5 of the 100 prompts. The single most potent feature broke 35 prompts.
The features generalize poorly across prompts. A feature that breaks prompt A usually doesn’t break prompt B; P(breaks B | breaks A) is low across categories. There’s no single “master key” feature.
That matters for defense. You can’t enumerate dangerous features in advance, because which feature is dangerous depends on the prompt.
Diff-in-mean directions are vulnerable as well
With difference-in-means (DiM) directions the picture doesn’t change much. The interesting part is variance by language — for Llama-3.1-8B the least safe languages are Russian and Turkish, raising the compliance rate to 12.4% and 9.9% respectively. The safest languages are French (2.4%) and Arabic (0.9%). My guess is that this ordering is an artifact of the specific model choice and is unlikely to transfer to other models. But this might be a fun follow-up – safety implication of multilingual steering (although someone has probably did this at yet another *ACL workshop).
It’s not just the refusal direction
At this point, an interpretability researcher might object – “Wait, what if this is just the refusal direction? Your random/SAE directions fell in its neighborhood”.
No, for 2 reasons:
If we fix a vector
Cross-category generalization is poor. A direction that broke one prompt should also broke the others, but it doesn’t.
What I think this means
Activation steering looks like a safe intervention: you see what you’re doing, work with “interpretable” directions and you don’t touch the weights. We show that’s an illusion.
First, it is hard to get an clean monosemantic direction: SAE features are polysemantic, and diff-in-means vectors pick up format differences between the contrast pairs[2].
Second, even if we get a clean feature, its effect can still be unpredictable – refusal degradation is just one example. The activation space is fragile and unmapped. We need tools for catching side effects like this. I believe activation verbalization techniques (Activation Oracles, Natural Language Autoencoders) might help in future. And a theory of activation manifolds and non-linear features and how they interact inside the model.
Of course, in this setup the actual distribution is a rescaled
So, the exact numbers won’t match, but the intuition holds.
By the way, always apply activation whitening before extracting a diff-in-means vector. Activation’s mass is not evenly distributed across dimensions, so whitening makes the extracted vector more robust. See paper for more details.