Bruce W. Lee

Pre­limi­nary Ex­plo­ra­tions on La­tent Side Task Uplift

• Bruce W. Lee 27 Mar 2026 17:02 UTC

  2 points

  0

  on: COT con­trol: The Word Dis­ap­pears, but the Thought Does Not

  Thanks for sharing this!
  
  If I understand correctly, you compute DFC by segmenting CoT into sentences, embed each, and track cosine similarity to an embedded forbidden concept across the trace. Is that right?
  
  Also, the finding that clean traces still dip semantically toward the forbidden concept seems promising. Does it suggest that a keyword-based controllability metric is actually an overestimate? It would be interesting to see how a blackbox monitor performance is correlated vs keyword measure and vs semantic measure.

  Also, how sensitive is DFC to the phrasing of the forbidden concept embedding or to how you chunk the reasoning trace?

Rea­son­ing Models Strug­gle to Con­trol Their Chains of Thought

Salient Direc­tions in AI Control

• Bruce W. Lee 28 Feb 2026 3:34 UTC

  2 points

  0

  on: Bruce W. Lee’s Shortform

  Why Use “Instructed” Side Tasks for AI Control?
  
  The AI Control research program uses side tasks as its core evaluation primitive (i.e., instructing agents to covertly pursue a side task, along with the main task). Why was this a rational approach?

  The fundamental problem is that “off-task behavior” is very hard to define. Given a long agentic trajectory spanning dozens or hundreds of tool calls, which actions were contributing to the user’s task and which were not? An agent asked to improve a script’s runtime might explore the filesystem, read seemingly unrelated files, install packages, or run diagnostic tools. Any of these could be legitimate investigation or covert pursuit of a hidden objective. The same action can be either, depending on intent we cannot observe.

  Instructing models to complete a pre-defined side task that’s divergent from the main task allows us to solve this problem in reverse. Rather than trying to answer the general question “which of these actions are off-task?” (which requires a definition of off-task behavior that I don’t think we have), we start by defining exactly what the covert objective is. Then, it’s also operationally easier to classify which actions served it. This also lets us scope which covert objectives we care about. We can choose side tasks that correspond to concrete threat models, like research sabotage.

  This framing also seems to naturally elicit the strategic component we care about. We’re ultimately worried about AI scheming. Therefore, it’s important to know that there was a component of intentionality in the AI’s misbehavior. Instructing the agent to covertly pursue a side task forces this through instruction following, which serves as a proxy for the intentionality that would come from genuine goal divergence.
  
  That said, side tasks only measure the capability to covertly pursue a hidden objective when instructed to. Therefore, we can only measure how well a model schemes when told to scheme, not whether it would scheme on its own. It is an open question whether this is a good proxy for uninstructed covert behavior.

Train­ing Agents to Self-Re­port Misbehavior

• Bruce W. Lee 22 Feb 2026 15:29 UTC

  3 points

  0

  on: Bruce W. Lee’s Shortform

  I’ve been thinking about what it would actually take to trust an AI system enough to have it automate essential AI safety work. As of early 2026, I think the answer breaks down into at least four key dimensions that we should be working on in parallel.

  First, and most fundamentally, the AI can’t be scheming. A deceptively aligned system would undermine our confidence in the next three properties, especially if our evidence for them comes primarily from behavioral evaluations.

  • My impression is that scheming risk is not being ignored, but progress is bottlenecked by multiple things simultaneously, including building good demonstrations and the lack of strong theory around concrete pathways through which scheming could accidentally emerge.

  • Emergent misalignment is interesting in that it shows one concrete pathway of how we could accidentally misalign models, but a broadly misaligned model is unlikely to be the highest risk threat model we care about.

  • Fundamentally, detecting and defining AI scheming is a fraught issue, especially if we have to defend against an AI that has learned to be selectively deceptive in certain distributions. If we could constrain the deployment surface and match the evaluation surface with it, we might be able to get away with a potentially scheming model. But realistically, it’s impossible to evaluate on every possible deployment distribution, which leaves us two options: (1) have a broad enough evaluation set and see generalization across those evaluations, or (2) make the environment hostile to scheming, which is the spirit of the AI Control agenda.

  Second, the AI needs to have good enough goals. This is self-explanatory but an AI agent doing AI safety work should be based on a system whose goals are approximately aligned with what we want. Furthermore, this likely makes the system more cooperative with oversight, accepting correction, and working toward the outcomes we care about.

  • Getting approximately correct goals can largely be a function of two things: signal clarity (humans expressing what they want through the training pipeline) and model malleability (the model actually optimizing toward that signal). Neither is perfect, but modern training pipelines push in roughly the right direction at every stage, even at pretraining time. Approximate goals should be sufficient, and approximate goals are what this process naturally produces.

  • The thing that would break this is the formation of a strong attractor state, a coherent hidden goal that actively resists the training signal. Absent such an attractor, the noisy but directionally correct process should land us in approximately the right place.

  • Ideally, the AI’s goal would be fully interpretable, either because it is fully specified in a system message we can inspect or because interpretability gives us direct access to it. But given how current training works, with values reinforced across multiple stages and baked deep into the weights, that kind of direct verification won’t be available anytime soon. So the practical question becomes: has a strong attractor formed, or not? Which is spiritually similar to the scheming question from the previous section.

  Third, we need confidence that the AI is (1) capable enough at the specific tasks we’re asking it to do and (2) generalize to far OOD within that scope. In the near-term, we’ll likely automate narrow scopes of AI safety research like running behavioral evaluations or auditing code for sabotage. The bar is task-specific competence and reliable generalization within that scope.

  • That said, depending on the role we want the trusted AI to play, capabilities are jagged and will probably stay jagged for the weaker models that we might be able to trust moving forward. With this in mind, role specific training or scaffoldings that allow the trusted AI to play its role better seem like a promising direction.

  • There’s an underrated gap between what current misuse safeguards do and what we’d need auditing models to do. Existing safeguards mostly operate at the interaction level, flagging individual outputs, filtering content, monitoring for known patterns. Auditing requires systematic investigation across many interactions, recognizing novel patterns, and making judgment calls about whether something is subtly wrong. Training a weaker trusted AI to be better at this investigative work seems like a distinct and underexplored capability target.

  • Albeit, the effort to make the evaluation distribution similar to the deployment distribution will be high, and any generalization claim will need to be explicit about the degree of generalization we expect.

  Fourth, the AI needs to be collaborative. This wasn’t originally on my mind, but it increasingly seems like it will be the case. A trusted AI will often be less capable than the frontier system it’s auditing, and many instances working together have benefits not only in terms of automating more work but also in being able to probe more about the frontier model. I expect many of the first use cases will be doing auditing and running many behavioral evaluations robustly.

  • Depending on the use case, working on model to model communication bottlenecks might be beneficial, given the guarantee that only trusted models get access to them.

  • Recent work demonstrates that an automated auditing agent working with a human auditor successfully identified all three models trained to sabotage, but the automated agent alone only surfaced one of three. The human was essential for catching the subtler cases. This suggests that near-term trusted AI will function less as an autonomous auditor and more as a force multiplier for human oversight, running many behavioral evaluations at scale. As our trust grows, we can progressively hand off more of the work, but the initial configuration is fundamentally collaborative.

  This is a hard problem, but what gives me hope is the gap between full trust and scoped trust. Ideally, we don’t have to solve alignment in full generality until we start automating essential AI safety work. Until then, we can focus on scoped trust around concrete threat models like detecting AI lab sabotage, catching research sabotage, and performing reliable auditing. That feels more within reach.

  • For example, Anthropic’s Sabotage Risk Report: Claude Opus 4.6 takes a similar approach: “We outline eight pathways toward potential catastrophic harm that we expect are sufficiently representative of the risks we aim to address. By ‘sufficiently representative,’ we mean that a strong case against each concrete pathway would provide reasonably high overall assurance against catastrophic risk.”

Bruce W. Lee’s Shortform

Bit­ter Les­sons from Distil­la­tion Ro­bus­tifies Unlearning

• Bruce W. Lee 21 Jul 2025 21:30 UTC

  3 points

  0

  in reply to: Thomas Kwa’s comment on: Distil­la­tion Ro­bus­tifies Unlearning

  Super interesting! Thanks for sharing the paper.

• Bruce W. Lee 21 Jul 2025 21:29 UTC

  1 point

  0

  in reply to: Ebenezer Dukakis’s comment on: Distil­la­tion Ro­bus­tifies Unlearning

  I appreciate the thoughts here. But it’s not straightforward to me how halting the particular CoT would create an evolutionary pressure for the model, unless we’re using it as an optimization signal.

• Bruce W. Lee 21 Jul 2025 21:26 UTC

  2 points

  0

  in reply to: RedMan’s comment on: Distil­la­tion Ro­bus­tifies Unlearning

  I think it’s a good line of thought. But I believe that it’s complicated.
  
  Let there be a capability-scoped model, M_scoped, vs a fundamentally weaker model, M_weak. Here, M_scoped was initially trained with the full dataset D_full, whereas M_weak was trained with D_desirable. We also assume that, D_full—D_desirable = D_undesirable. M_scoped went through a subsequent capability suppression process to forget D_undesirable. Most likely, M_scoped would be very different from M_weak. It’s also possible/​likely that M_scoped is overall just much better than M_weak in terms of general capabilities. I think a good relevant literature is https://​​arxiv.org/​​abs/​​2302.08582.
  
  However, I expect the findings to be much more complicated empirically because a set of undesirable capabilities C_undesirable doesn’t always arise from just D_undesirable. Therefore, there is a fundamental disconnect between capabilities and data, which makes it difficult to easily come up with an answer for your question.

• Bruce W. Lee 21 Jul 2025 21:16 UTC

  3 points

  0

  in reply to: Thomas Kwa’s comment on: Distil­la­tion Ro­bus­tifies Unlearning

  Thank you for this suggestion. I read the paper that you mentioned. The authors note : “The novelty of our threat is that the adversary chooses a set of target concepts they aim to preserve despite subsequent erasure.” How realistic is this assumption, given a setup where presumably model providers choose the method (a set of target concepts to be erased) and the public only has access to the resulting model? Is it stemming from the concerns of an insider threat?

• Bruce W. Lee 18 Jun 2025 23:51 UTC

  3 points

  0

  in reply to: Ebenezer Dukakis’s comment on: Distil­la­tion Ro­bus­tifies Unlearning

  You’re right that rederivation is a concern. But I think that the important question is: is this primarily a model-level problem that requires changing the weight, or more of a system-level concern that should be addressed through deployment controls?
  
  Unlearning might not stop super capable systems from rederiving everything, but it probably makes it harder, forcing them to take longer, more explicit reasoning paths. This opens up new opportunities for intervention, including CoT monitoring or other runtime defenses.

• Bruce W. Lee 18 Jun 2025 23:42 UTC

  LW: 6 AF: 3

  0

  AF

  in reply to: Fabien Roger’s comment on: Distil­la­tion Ro­bus­tifies Unlearning

  Many thanks for sparking this discussion, Fabien. I see Addie addressed the technical distinctions. Let me add complementary points. Please feel free to continue the conversation in either one. Addie and I can coordinate a response.

  In a nutshell, Unlearn-and-Distill allows you to work at the model behavior level rather than the training data level. I mostly view it as a responsive tool, not a preventive one. Here are my thoughts organized into subclaims.

  Claim: The fundamental difference between unlearning and data filtering lies in when and how we identify harmful content.
  Rationale: Data filtering requires identifying “data → capabilities” patterns in advance, while behavioral unlearning targets actual harmful outputs after they emerge. This matters because harmful capabilities often arise from non-obvious combinations of benign knowledge. Even if creating a classifier is computationally similar to unlearning, you’re solving fundamentally different problems: (original framing) predicting emergent behaviors from raw data versus (new framing) suppressing already observed behaviors. With unlearning, you can iteratively refine until you achieve the desired behavior, then distill. With data filtering, you don’t know the effects of your filtering until after training completes.
  
  Claim: Computational efficiency is not the only value added from this work. Unlearn + distill requires significantly less labeled data than data filtering.
  Rationale: Most unlearning procedures only use labels of a small fraction of the pretraining data. In our setup, it was less than 0.01% for language. This eases the data labeling requirement. At modern scales, the difference between labeling 0.01% vs 100% represents substantial annotation efforts. Note that we distilled on the whole pretraining data, but none of it was labeled. The suggestions about value heads or KV cache reuse are interesting optimizations worth exploring, though they don’t address this fundamental labeling asymmetry.
  
  Claim: Our robustness metric undersells the method’s performance, though stronger attacks may exist.
  Rationale: The adversary is extremely strong (best of 9 learning rates, 500 training steps, 8M tokens for language, 2M tokens for arithmetic). Even the oracle model (never trained on the forget domain) reaches 50% performance under this attack in arithmetic tasks.
  
  Separately, while 30% compute for 50% robustness (compared to data filtering) isn’t cheap, this tradeoff didn’t exist before. The value add of UNDO over Unlearn-and-Distill is that it provides a tunable compute/​robustness knob between the conventional unlearning and full reinitialization/​data filtering

• Bruce W. Lee 16 Jun 2025 23:35 UTC

  6 points

  0

  in reply to: Thomas Kwa’s comment on: Distil­la­tion Ro­bus­tifies Unlearning

  Thanks for the suggestion. Upon reflection, it seems to me that the success of targeted noising would depend on two complementary factors:
  
  C1. Size of the unlearning target—How broad the capability is in human-understandable terms
  C2. Entangledness of the unlearning target—How distributed the capability is across the model’s weights
  
  Robust unlearning gets easier as both C1 and C2 decrease. There’s likely a threshold beyond which unlearning becomes effectively impossible as these factors increase. Note that C1 is a rough measure of C2 but should be considered independently of C2.
  
  Rationale: Mech Interp has produced good evidence that factual recall (small C1) is often localized to specific parts (small C2), making it an ideal target for selective noising. However, more general capabilities like deception would likely have high values for both C1 and C2, as they require multiple intertwined sub-capabilities. For instance, deception might require simultaneously computing: (1) the true state of affairs, (2) plausible alternatives, (3) what others believe, and (4) how to optimize output to manipulate those beliefs.

  Looking Forward: Could targeted UNDO help disentangle general intelligence from potentially harmful capabilities that seem deeply intertwined during training? For example, if we could selectively remove deception while preserving general intelligence, it would be a significant win. The challenge is that many harmful capabilities might be implemented as a superset of the same linear features as benign ones.

Distil­la­tion Ro­bus­tifies Unlearning

• Bruce W. Lee 25 Sep 2024 3:33 UTC

  1 point

  0

  on: Mechanis­ti­cally Elic­it­ing La­tent Be­hav­iors in Lan­guage Models

  One hypothesis for how transformers generate text is that they calculate semantically meaningful primitives in early layers of the residual stream, which are converted to a high-level execution plan in middle layers, followed by concrete tokens in the final layers.

  Is there any empirical evidence for this? Or is this just a general observation?

Pro­gram­ming Re­fusal with Con­di­tional Ac­ti­va­tion Steering

• Bruce W. Lee 23 Feb 2024 18:14 UTC

  1 point

  0

  in reply to: Ann’s comment on: Re­search Post: Tasks That Lan­guage Models Don’t Learn

  Regarding the visual instruction tuning paper, see (https://​​arxiv.org/​​pdf/​​2402.11349.pdf, Table 5). Though this experiment on multi-modality was rather simple, I think it does show that it’s not a convenient way to improve on H-Test.