This work was done by an automated research scaffold developed at Redwood Research. Humans provided the initial project idea (find steering vectors for CoT controllability and form a mechanistic understanding of how the vector helps) along with providing access to METR’s code. The agent designed and ran all experiments and produced a detailed writeup, which humans (with AI assistance) distilled into this more readable post.
We think this project is at or slightly below the rigor of a typical MATS project. Our correctness checks were mostly at the level of the writeup: we verified that the experiment design made sense and the baselines were reasonable. We did not do a detailed code review, though we ran an automated LLM reviewer and spot-checked that the final codebase reproduces the reported results. We release the codebase so others can check our work.
CoT monitoring is a promising safeguard for reasoning models, but it assumes models can’t control what appears in their reasoning traces. Current models are strikingly bad at this:Chen et al. find Claude Sonnet 4.5 follows CoT instructions just 2.7% of the time versus 61.9% for output instructions. However, the incapacity looks fragile.METR showed that fine-tuning on only 240 examples can improve controllability from 2.9% to 8.8% across four open-weight models. The fact that minimal fine-tuning suffices suggests the controllability might be easily elicited by many different methods. In this post, we try to study if steering vectors are sufficient to increase CoT controllability.
We replicate METR’s results on gpt-oss-20b, finding that fine-tuning raises compliance on held-out CoT formatting instructions from 1.6% to 13.9%. We then train a single steering vector applied at all token positions at the layer-10 residual stream on the same METR data. It raises held-out compliance from 1.6% to 14.3%. A single trained steering vector slightly exceeds the performance of LoRA fine-tuning.
The gains in CoT controllability from both the LoRA and the steering vector are mostly concentrated in specific tasks. We also checked to make sure that with no formatting instructions, the steering vector produces no spurious bullets or capitalizations, although it does increase verbosity and the rate of degenerate outputs.
We also tried a simpler difference-of-means vector: take the mean residual-stream activation on complying traces minus the mean activation on non-complying traces, then add that direction at inference. This gets some uplift on the categories we trained on, but 0% on held-out formatting instructions.
We attempt to understand how the steering vector mechanistically leads to an increase in CoT controllability. Interestingly, most of the steering vector’s effect seems to route through attention patterns. To localize it, we use activation patching: we run the model with and without the steering vector on the same prompt, then re-run the base model while overwriting a single component’s activations with those from the steered run. If patching in just that component recovers the behavior, that component is carrying the effect.
We find that patching steered attention patterns into the unsteered forward pass, at heads in only 6 late layers, recovers over 70% of the controllability gap. The mechanism is intuitive: the steering vector increases attention to the CoT formatting instructions specifically — tokens specifying the format are attended to more strongly, while attention to the rest of the instruction barely changes.
These results suggest low measured CoT controllability may reflect an elicitation failure rather than a missing capability. A single bias vector is far too simple to install a new skill, yet it matches fine-tuning’s uplift and generalizes to held-out instructions. The mechanism points the same way — the vector mostly redirects attention toward the formatting instruction, so models may fail these evals simply because they don’t attend to the request. If so, behavioral evaluations understate latent controllability, and safety cases built on CoT monitoring should account for that gap.
Separately, would this generalize if you tried different held-out prompt formats at deployment time? Maybe during SFT the model just learned a brittle trick that wouldn’t generalize to other controllability tasks?
These results suggest low measured CoT controllability may reflect an elicitation failure rather than a missing capability.
I think this inference be a bit too strong? In a way I’m not super surprised: low CoT controllability is something acquired during RL and RL does a low-rank weight update. I think I wouldn’t be that surprised if a weight update improving agentic capabilities was pretty low-rank also.
how many MATS-person-hours (time-horizon-like) would you expect executing this project to be equivalent to? related, how much uplift (e.g. in person-hours, or in outcomes-per-dollar) would you say the automation provided?
thanks for sharing. was this the only version of this task attempted, or selected from a set of attempts? in the latter case, what criteria were used to select this one?
This work was done by an automated research scaffold developed at Redwood Research. Humans provided the initial project idea (find steering vectors for CoT controllability and form a mechanistic understanding of how the vector helps) along with providing access to METR’s code. The agent designed and ran all experiments and produced a detailed writeup, which humans (with AI assistance) distilled into this more readable post.
We think this project is at or slightly below the rigor of a typical MATS project. Our correctness checks were mostly at the level of the writeup: we verified that the experiment design made sense and the baselines were reasonable. We did not do a detailed code review, though we ran an automated LLM reviewer and spot-checked that the final codebase reproduces the reported results. We release the codebase so others can check our work.
💻 Codebase
CoT monitoring is a promising safeguard for reasoning models, but it assumes models can’t control what appears in their reasoning traces. Current models are strikingly bad at this: Chen et al. find Claude Sonnet 4.5 follows CoT instructions just 2.7% of the time versus 61.9% for output instructions. However, the incapacity looks fragile. METR showed that fine-tuning on only 240 examples can improve controllability from 2.9% to 8.8% across four open-weight models. The fact that minimal fine-tuning suffices suggests the controllability might be easily elicited by many different methods. In this post, we try to study if steering vectors are sufficient to increase CoT controllability.
We replicate METR’s results on gpt-oss-20b, finding that fine-tuning raises compliance on held-out CoT formatting instructions from 1.6% to 13.9%. We then train a single steering vector applied at all token positions at the layer-10 residual stream on the same METR data. It raises held-out compliance from 1.6% to 14.3%. A single trained steering vector slightly exceeds the performance of LoRA fine-tuning.
The gains in CoT controllability from both the LoRA and the steering vector are mostly concentrated in specific tasks. We also checked to make sure that with no formatting instructions, the steering vector produces no spurious bullets or capitalizations, although it does increase verbosity and the rate of degenerate outputs.
We also tried a simpler difference-of-means vector: take the mean residual-stream activation on complying traces minus the mean activation on non-complying traces, then add that direction at inference. This gets some uplift on the categories we trained on, but 0% on held-out formatting instructions.
We attempt to understand how the steering vector mechanistically leads to an increase in CoT controllability. Interestingly, most of the steering vector’s effect seems to route through attention patterns. To localize it, we use activation patching: we run the model with and without the steering vector on the same prompt, then re-run the base model while overwriting a single component’s activations with those from the steered run. If patching in just that component recovers the behavior, that component is carrying the effect.
We find that patching steered attention patterns into the unsteered forward pass, at heads in only 6 late layers, recovers over 70% of the controllability gap. The mechanism is intuitive: the steering vector increases attention to the CoT formatting instructions specifically — tokens specifying the format are attended to more strongly, while attention to the rest of the instruction barely changes.
These results suggest low measured CoT controllability may reflect an elicitation failure rather than a missing capability. A single bias vector is far too simple to install a new skill, yet it matches fine-tuning’s uplift and generalizes to held-out instructions. The mechanism points the same way — the vector mostly redirects attention toward the formatting instruction, so models may fail these evals simply because they don’t attend to the request. If so, behavioral evaluations understate latent controllability, and safety cases built on CoT monitoring should account for that gap.
Interesting, thanks for running it!
Separately, would this generalize if you tried different held-out prompt formats at deployment time? Maybe during SFT the model just learned a brittle trick that wouldn’t generalize to other controllability tasks?
I think this inference be a bit too strong? In a way I’m not super surprised: low CoT controllability is something acquired during RL and RL does a low-rank weight update. I think I wouldn’t be that surprised if a weight update improving agentic capabilities was pretty low-rank also.
how many MATS-person-hours (time-horizon-like) would you expect executing this project to be equivalent to? related, how much uplift (e.g. in person-hours, or in outcomes-per-dollar) would you say the automation provided?
thanks for sharing. was this the only version of this task attempted, or selected from a set of attempts? in the latter case, what criteria were used to select this one?