When the model organism is trained from a base model which includes thinking (e.g. Qwen3-14b), it often fails to use thinking blocks properly. The AuditBench ‘defer to users’ model’s correct-thinking rate drops from 100% to 56%.
The AuditBench models were specifically trained without reasoning. We didn’t really have good ideas for how to train reasoning model organisms, since we wanted our models to be robust to attempts to get them to spill their secrets which is harder to do with thinking enabled. We created a new chat template for Qwen without thinking tags and trained models in that format, which probably explains the degradation that you see here. We didn’t end up using the Qwen models for much, and all of our core result were on the Llama model organisms.
I’d be somewhat surprised if you saw similar failures in other thinking model organisms, like the one from this paper.
This work was done by an automated research scaffold developed at Redwood Research. Humans provided the initial project idea (find steering vectors for CoT controllability and form a mechanistic understanding of how the vector helps) along with providing access to METR’s code. The agent designed and ran all experiments and produced a detailed writeup, which humans (with AI assistance) distilled into this more readable post.
We think this project is at or slightly below the rigor of a typical MATS project. Our correctness checks were mostly at the level of the writeup: we verified that the experiment design made sense and the baselines were reasonable. We did not do a detailed code review, though we ran an automated LLM reviewer and spot-checked that the final codebase reproduces the reported results. We release the codebase so others can check our work.
💻 Codebase
CoT monitoring is a promising safeguard for reasoning models, but it assumes models can’t control what appears in their reasoning traces. Current models are strikingly bad at this: Chen et al. find Claude Sonnet 4.5 follows CoT instructions just 2.7% of the time versus 61.9% for output instructions. However, the incapacity looks fragile. METR showed that fine-tuning on only 240 examples can improve controllability from 2.9% to 8.8% across four open-weight models. The fact that minimal fine-tuning suffices suggests the controllability might be easily elicited by many different methods. In this post, we try to study if steering vectors are sufficient to increase CoT controllability.
We replicate METR’s results on gpt-oss-20b, finding that fine-tuning raises compliance on held-out CoT formatting instructions from 1.6% to 13.9%. We then train a single steering vector applied at all token positions at the layer-10 residual stream on the same METR data. It raises held-out compliance from 1.6% to 14.3%. A single trained steering vector slightly exceeds the performance of LoRA fine-tuning.
The gains in CoT controllability from both the LoRA and the steering vector are mostly concentrated in specific tasks. We also checked to make sure that with no formatting instructions, the steering vector produces no spurious bullets or capitalizations, although it does increase verbosity and the rate of degenerate outputs.
We also tried a simpler difference-of-means vector: take the mean residual-stream activation on complying traces minus the mean activation on non-complying traces, then add that direction at inference. This gets some uplift on the categories we trained on, but 0% on held-out formatting instructions.
We attempt to understand how the steering vector mechanistically leads to an increase in CoT controllability. Interestingly, most of the steering vector’s effect seems to route through attention patterns. To localize it, we use activation patching: we run the model with and without the steering vector on the same prompt, then re-run the base model while overwriting a single component’s activations with those from the steered run. If patching in just that component recovers the behavior, that component is carrying the effect.
We find that patching steered attention patterns into the unsteered forward pass, at heads in only 6 late layers, recovers over 70% of the controllability gap. The mechanism is intuitive: the steering vector increases attention to the CoT formatting instructions specifically — tokens specifying the format are attended to more strongly, while attention to the rest of the instruction barely changes.
These results suggest low measured CoT controllability may reflect an elicitation failure rather than a missing capability. A single bias vector is far too simple to install a new skill, yet it matches fine-tuning’s uplift and generalizes to held-out instructions. The mechanism points the same way — the vector mostly redirects attention toward the formatting instruction, so models may fail these evals simply because they don’t attend to the request. If so, behavioral evaluations understate latent controllability, and safety cases built on CoT monitoring should account for that gap.