Bruce W. Lee

Karma: 521

Bruce W. Lee 27 Mar 2026 17:02 UTC
2 points
0
on: COT control: The Word Disappears, but the Thought Does Not
Thanks for sharing this!

If I understand correctly, you compute DFC by segmenting CoT into sentences, embed each, and track cosine similarity to an embedded forbidden concept across the trace. Is that right?

Also, the finding that clean traces still dip semantically toward the forbidden concept seems promising. Does it suggest that a keyword-based controllability metric is actually an overestimate? It would be interesting to see how a blackbox monitor performance is correlated vs keyword measure and vs semantic measure.
Also, how sensitive is DFC to the phrasing of the forbidden concept embedding or to how you chunk the reasoning trace?

Bruce W. Lee 28 Feb 2026 3:34 UTC
2 points
0
on: Bruce W. Lee’s Shortform
Why Use “Instructed” Side Tasks for AI Control?

The AI Control research program uses side tasks as its core evaluation primitive (i.e., instructing agents to covertly pursue a side task, along with the main task). Why was this a rational approach?
The fundamental problem is that “off-task behavior” is very hard to define. Given a long agentic trajectory spanning dozens or hundreds of tool calls, which actions were contributing to the user’s task and which were not? An agent asked to improve a script’s runtime might explore the filesystem, read seemingly unrelated files, install packages, or run diagnostic tools. Any of these could be legitimate investigation or covert pursuit of a hidden objective. The same action can be either, depending on intent we cannot observe.
Instructing models to complete a pre-defined side task that’s divergent from the main task allows us to solve this problem in reverse. Rather than trying to answer the general question “which of these actions are off-task?” (which requires a definition of off-task behavior that I don’t think we have), we start by defining exactly what the covert objective is. Then, it’s also operationally easier to classify which actions served it. This also lets us scope which covert objectives we care about. We can choose side tasks that correspond to concrete threat models, like research sabotage.
This framing also seems to naturally elicit the strategic component we care about. We’re ultimately worried about AI scheming. Therefore, it’s important to know that there was a component of intentionality in the AI’s misbehavior. Instructing the agent to covertly pursue a side task forces this through instruction following, which serves as a proxy for the intentionality that would come from genuine goal divergence.

That said, side tasks only measure the capability to covertly pursue a hidden objective when instructed to. Therefore, we can only measure how well a model schemes when told to scheme, not whether it would scheme on its own. It is an open question whether this is a good proxy for uninstructed covert behavior.

Bruce W. Lee 22 Feb 2026 15:29 UTC
3 points
0
on: Bruce W. Lee’s Shortform
I’ve been thinking about what it would actually take to trust an AI system enough to have it automate essential AI safety work. As of early 2026, I think the answer breaks down into at least four key dimensions that we should be working on in parallel.
First, and most fundamentally, the AI can’t be scheming. A deceptively aligned system would undermine our confidence in the next three properties, especially if our evidence for them comes primarily from behavioral evaluations.
- My impression is that scheming risk is not being ignored, but progress is bottlenecked by multiple things simultaneously, including building good demonstrations and the lack of strong theory around concrete pathways through which scheming could accidentally emerge.
- Emergent misalignment is interesting in that it shows one concrete pathway of how we could accidentally misalign models, but a broadly misaligned model is unlikely to be the highest risk threat model we care about.
- Fundamentally, detecting and defining AI scheming is a fraught issue, especially if we have to defend against an AI that has learned to be selectively deceptive in certain distributions. If we could constrain the deployment surface and match the evaluation surface with it, we might be able to get away with a potentially scheming model. But realistically, it’s impossible to evaluate on every possible deployment distribution, which leaves us two options: (1) have a broad enough evaluation set and see generalization across those evaluations, or (2) make the environment hostile to scheming, which is the spirit of the AI Control agenda.
Second, the AI needs to have good enough goals. This is self-explanatory but an AI agent doing AI safety work should be based on a system whose goals are approximately aligned with what we want. Furthermore, this likely makes the system more cooperative with oversight, accepting correction, and working toward the outcomes we care about.
- Getting approximately correct goals can largely be a function of two things: signal clarity (humans expressing what they want through the training pipeline) and model malleability (the model actually optimizing toward that signal). Neither is perfect, but modern training pipelines push in roughly the right direction at every stage, even at pretraining time. Approximate goals should be sufficient, and approximate goals are what this process naturally produces.
- The thing that would break this is the formation of a strong attractor state, a coherent hidden goal that actively resists the training signal. Absent such an attractor, the noisy but directionally correct process should land us in approximately the right place.
- Ideally, the AI’s goal would be fully interpretable, either because it is fully specified in a system message we can inspect or because interpretability gives us direct access to it. But given how current training works, with values reinforced across multiple stages and baked deep into the weights, that kind of direct verification won’t be available anytime soon. So the practical question becomes: has a strong attractor formed, or not? Which is spiritually similar to the scheming question from the previous section.
Third, we need confidence that the AI is (1) capable enough at the specific tasks we’re asking it to do and (2) generalize to far OOD within that scope. In the near-term, we’ll likely automate narrow scopes of AI safety research like running behavioral evaluations or auditing code for sabotage. The bar is task-specific competence and reliable generalization within that scope.
- That said, depending on the role we want the trusted AI to play, capabilities are jagged and will probably stay jagged for the weaker models that we might be able to trust moving forward. With this in mind, role specific training or scaffoldings that allow the trusted AI to play its role better seem like a promising direction.
- There’s an underrated gap between what current misuse safeguards do and what we’d need auditing models to do. Existing safeguards mostly operate at the interaction level, flagging individual outputs, filtering content, monitoring for known patterns. Auditing requires systematic investigation across many interactions, recognizing novel patterns, and making judgment calls about whether something is subtly wrong. Training a weaker trusted AI to be better at this investigative work seems like a distinct and underexplored capability target.
- Albeit, the effort to make the evaluation distribution similar to the deployment distribution will be high, and any generalization claim will need to be explicit about the degree of generalization we expect.
Fourth, the AI needs to be collaborative. This wasn’t originally on my mind, but it increasingly seems like it will be the case. A trusted AI will often be less capable than the frontier system it’s auditing, and many instances working together have benefits not only in terms of automating more work but also in being able to probe more about the frontier model. I expect many of the first use cases will be doing auditing and running many behavioral evaluations robustly.
- Depending on the use case, working on model to model communication bottlenecks might be beneficial, given the guarantee that only trusted models get access to them.
- Recent work demonstrates that an automated auditing agent working with a human auditor successfully identified all three models trained to sabotage, but the automated agent alone only surfaced one of three. The human was essential for catching the subtler cases. This suggests that near-term trusted AI will function less as an autonomous auditor and more as a force multiplier for human oversight, running many behavioral evaluations at scale. As our trust grows, we can progressively hand off more of the work, but the initial configuration is fundamentally collaborative.
This is a hard problem, but what gives me hope is the gap between full trust and scoped trust. Ideally, we don’t have to solve alignment in full generality until we start automating essential AI safety work. Until then, we can focus on scoped trust around concrete threat models like detecting AI lab sabotage, catching research sabotage, and performing reliable auditing. That feels more within reach.
- For example, Anthropic’s Sabotage Risk Report: Claude Opus 4.6 takes a similar approach: “We outline eight pathways toward potential catastrophic harm that we expect are sufficiently representative of the risks we aim to address. By ‘sufficiently representative,’ we mean that a strong case against each concrete pathway would provide reasonably high overall assurance against catastrophic risk.”

Bruce W. Lee 21 Jul 2025 21:30 UTC
3 points
0
in reply to: Thomas Kwa’s comment on: Distillation Robustifies Unlearning
Super interesting! Thanks for sharing the paper.

Bruce W. Lee 21 Jul 2025 21:29 UTC
1 point
0
in reply to: Ebenezer Dukakis’s comment on: Distillation Robustifies Unlearning
I appreciate the thoughts here. But it’s not straightforward to me how halting the particular CoT would create an evolutionary pressure for the model, unless we’re using it as an optimization signal.

Bruce W. Lee 21 Jul 2025 21:26 UTC
2 points
0
in reply to: RedMan’s comment on: Distillation Robustifies Unlearning
I think it’s a good line of thought. But I believe that it’s complicated.

Let there be a capability-scoped model, M_scoped, vs a fundamentally weaker model, M_weak. Here, M_scoped was initially trained with the full dataset D_full, whereas M_weak was trained with D_desirable. We also assume that, D_full—D_desirable = D_undesirable. M_scoped went through a subsequent capability suppression process to forget D_undesirable. Most likely, M_scoped would be very different from M_weak. It’s also possible/likely that M_scoped is overall just much better than M_weak in terms of general capabilities. I think a good relevant literature is https://arxiv.org/abs/2302.08582.

However, I expect the findings to be much more complicated empirically because a set of undesirable capabilities C_undesirable doesn’t always arise from just D_undesirable. Therefore, there is a fundamental disconnect between capabilities and data, which makes it difficult to easily come up with an answer for your question.

Bruce W. Lee 21 Jul 2025 21:16 UTC
3 points
0
in reply to: Thomas Kwa’s comment on: Distillation Robustifies Unlearning
Thank you for this suggestion. I read the paper that you mentioned. The authors note : “The novelty of our threat is that the adversary chooses a set of target concepts they aim to preserve despite subsequent erasure.” How realistic is this assumption, given a setup where presumably model providers choose the method (a set of target concepts to be erased) and the public only has access to the resulting model? Is it stemming from the concerns of an insider threat?

Bruce W. Lee 18 Jun 2025 23:51 UTC
3 points
0
in reply to: Ebenezer Dukakis’s comment on: Distillation Robustifies Unlearning
You’re right that rederivation is a concern. But I think that the important question is: is this primarily a model-level problem that requires changing the weight, or more of a system-level concern that should be addressed through deployment controls?

Unlearning might not stop super capable systems from rederiving everything, but it probably makes it harder, forcing them to take longer, more explicit reasoning paths. This opens up new opportunities for intervention, including CoT monitoring or other runtime defenses.

Bruce W. Lee 18 Jun 2025 23:42 UTC
LW: 6 AF: 3
0
AF
in reply to: Fabien Roger’s comment on: Distillation Robustifies Unlearning
Many thanks for sparking this discussion, Fabien. I see Addie addressed the technical distinctions. Let me add complementary points. Please feel free to continue the conversation in either one. Addie and I can coordinate a response.
In a nutshell, Unlearn-and-Distill allows you to work at the model behavior level rather than the training data level. I mostly view it as a responsive tool, not a preventive one. Here are my thoughts organized into subclaims.
Claim: The fundamental difference between unlearning and data filtering lies in when and how we identify harmful content.
Rationale: Data filtering requires identifying “data → capabilities” patterns in advance, while behavioral unlearning targets actual harmful outputs after they emerge. This matters because harmful capabilities often arise from non-obvious combinations of benign knowledge. Even if creating a classifier is computationally similar to unlearning, you’re solving fundamentally different problems: (original framing) predicting emergent behaviors from raw data versus (new framing) suppressing already observed behaviors. With unlearning, you can iteratively refine until you achieve the desired behavior, then distill. With data filtering, you don’t know the effects of your filtering until after training completes.

Claim: Computational efficiency is not the only value added from this work. Unlearn + distill requires significantly less labeled data than data filtering.
Rationale: Most unlearning procedures only use labels of a small fraction of the pretraining data. In our setup, it was less than 0.01% for language. This eases the data labeling requirement. At modern scales, the difference between labeling 0.01% vs 100% represents substantial annotation efforts. Note that we distilled on the whole pretraining data, but none of it was labeled. The suggestions about value heads or KV cache reuse are interesting optimizations worth exploring, though they don’t address this fundamental labeling asymmetry.

Claim: Our robustness metric undersells the method’s performance, though stronger attacks may exist.
Rationale: The adversary is extremely strong (best of 9 learning rates, 500 training steps, 8M tokens for language, 2M tokens for arithmetic). Even the oracle model (never trained on the forget domain) reaches 50% performance under this attack in arithmetic tasks.

Separately, while 30% compute for 50% robustness (compared to data filtering) isn’t cheap, this tradeoff didn’t exist before. The value add of UNDO over Unlearn-and-Distill is that it provides a tunable compute/robustness knob between the conventional unlearning and full reinitialization/data filtering

Bruce W. Lee 16 Jun 2025 23:35 UTC
6 points
0
in reply to: Thomas Kwa’s comment on: Distillation Robustifies Unlearning
Thanks for the suggestion. Upon reflection, it seems to me that the success of targeted noising would depend on two complementary factors:

C1. Size of the unlearning target—How broad the capability is in human-understandable terms
C2. Entangledness of the unlearning target—How distributed the capability is across the model’s weights

Robust unlearning gets easier as both C1 and C2 decrease. There’s likely a threshold beyond which unlearning becomes effectively impossible as these factors increase. Note that C1 is a rough measure of C2 but should be considered independently of C2.

Rationale: Mech Interp has produced good evidence that factual recall (small C1) is often localized to specific parts (small C2), making it an ideal target for selective noising. However, more general capabilities like deception would likely have high values for both C1 and C2, as they require multiple intertwined sub-capabilities. For instance, deception might require simultaneously computing: (1) the true state of affairs, (2) plausible alternatives, (3) what others believe, and (4) how to optimize output to manipulate those beliefs.
Looking Forward: Could targeted UNDO help disentangle general intelligence from potentially harmful capabilities that seem deeply intertwined during training? For example, if we could selectively remove deception while preserving general intelligence, it would be a significant win. The challenge is that many harmful capabilities might be implemented as a superset of the same linear features as benign ones.

Bruce W. Lee 25 Sep 2024 3:33 UTC
1 point
0
on: Mechanistically Eliciting Latent Behaviors in Language Models
One hypothesis for how transformers generate text is that they calculate semantically meaningful primitives in early layers of the residual stream, which are converted to a high-level execution plan in middle layers, followed by concrete tokens in the final layers.
Is there any empirical evidence for this? Or is this just a general observation?

Bruce W. Lee 23 Feb 2024 18:14 UTC
1 point
0
in reply to: Ann’s comment on: Research Post: Tasks That Language Models Don’t Learn
Regarding the visual instruction tuning paper, see (https://arxiv.org/pdf/2402.11349.pdf, Table 5). Though this experiment on multi-modality was rather simple, I think it does show that it’s not a convenient way to improve on H-Test.

Bruce W. Lee 23 Feb 2024 17:06 UTC
1 point
0
in reply to: Ann’s comment on: Research Post: Tasks That Language Models Don’t Learn
Out of genuine curiosity, can you link to your sources?

Bruce W. Lee 23 Feb 2024 16:15 UTC
1 point
0
in reply to: gwern’s comment on: Research Post: Tasks That Language Models Don’t Learn
Thanks for the comment. I’ll get back to you sometime soon.

Before I come up with anything though, where are you getting to with your arguments? It would help me draft a better reply if I knew your ultimatum.

Bruce W. Lee 23 Feb 2024 15:37 UTC
1 point
0
in reply to: wesg’s comment on: Research Post: Tasks That Language Models Don’t Learn
I also want to point you to this (https://arxiv.org/abs/2402.11349, Appendix I, Figure 7, Last Page, “Blueberry?: From Reddit u/AwkwardIllustrator47, r/mkbhd: Was listening to the podcast. Can anyone explain why Chat GPT doesn’t know if R is in the word Blueberry?”). Large model failures on these task types were rather a widely observed phenomenon but with no empirical investigation.

Bruce W. Lee 23 Feb 2024 15:31 UTC
1 point
0
in reply to: p.b.’s comment on: Research Post: Tasks That Language Models Don’t Learn
About 1.) Agree with this duality argument.

About 2.) I’m aware of the type of tasks that suddenly increase in performance at a certain scale, but it is rather challenging to confirm assertions about the emergence of capabilities at certain model scales. If I made a claim like “it seems that emergence happens at 1TB model size like GPT-4”, it would be misleading as there are too many compound variables in play. However, it would also be a false belief to claim that absolutely nothing happens at such an astronomical model size.
Our paper’s stance, phrased carefully (and hopefully firmly), is that larger models from the same family (e.g., LLaMA 2 13B to LLaMA 2 70B) don’t automatically lead to better H-Test performance. In terms of understanding GPT-4 performance (Analysis: We Don’t Understand GPT-4), we agreed that we should be blunt about why GPT-4 is performing so well due to too many compound variables.
As for Claude, we refrained from speculating about scale since we didn’t observe its impact directly. Given the lack of transparency about model sizes from AI labs, and considering other models in our study that performed on par with Claude on benchmarks like MMLU, we can’t attribute Claude’s 60% accuracy solely to scale. Even if we view this accuracy as more than marginal improvement, it suggests that Claude is doing something distinct, resulting in a greater boost on H-Test compared to what one might expect from scaling effects on other benchmarks.
About 3.) Fine-tuning can indeed be effective for prompting models to memorize information. In our study, this approach served as a useful proxy for testing the models’ ability to learn from orthography-specific data, without yielding substantial performance improvements on H-Test.

Bruce W. Lee 23 Feb 2024 5:15 UTC
1 point
0
in reply to: wesg’s comment on: Research Post: Tasks That Language Models Don’t Learn
I appreciate this analysis. I’ll take more time to look into this and then get back to write a better reply.

Bruce W. Lee 23 Feb 2024 5:13 UTC
3 points
1
in reply to: gwern’s comment on: Research Post: Tasks That Language Models Don’t Learn
So to summarize your claim (check if I’m understanding correctly):

1. Character-level tokenization can lead to different results.
- My answer: Yes and No. But mostly no. H-Test is not just any set of character manipulation tasks.
- Explanation: Maybe some H-Test tasks can be affected by this. But how do you explain tasks like Repeated Word (one group has two repeated words) or End Punctuation (based on the location of the punctuation). Though this opinion is valid and is probably worthy of further investigation, it doesn’t disprove the full extent of our tests. Along similar lines, GPT-4 shows some of the most “jumping” performance improvement from GPT 3.5 in non-character-level tasks (Repeated Word: 0.505 → 0.98).

2. Scaling up will lead to better results. Since no other models tested were at the scale of GPT-4, that’s why they couldn’t solve H-Test.
- My answer: No but it would be interesting if this turned out to be true.
- Explanation: We tested 15 models from leading LLM labs before we arrived at our claim. If the H-Test was a “scaling task”, we would have observed at least some degree of performance improvement in other models like Luminous or LLaMA too. But no this was not the case. And the research that you linked doesn’t seem to devise a text-to-text setup to test this ability.

3. Memorization (aka more orthography-specific data) will lead to better results.
- My answer: No.
- Explanation: Our section 5 (Analysis: We Don’t Understand GPT-4) is in fact dedicated to disproving the claim that more orthography-specific data will help LLMs solve H-Test. In GPT-3.5-Turbo finetuning results on H-Test training set, we observed no significant improvement in performance. Before and after finetuning, the performance remains tightly centered around the random change baseline.

Bruce W. Lee 22 Feb 2024 23:02 UTC
1 point
0
in reply to: Denys Cherhykalo’s comment on: Research Post: Tasks That Language Models Don’t Learn
“image caption generation and video simulation currently used in Sora will partially correct some of these errors.” I’m in line with this idea.

Bruce W. Lee 22 Feb 2024 20:51 UTC
1 point
0
in reply to: jacopo’s comment on: Research Post: Tasks That Language Models Don’t Learn
I initially thought so until the GPT-4 results came back. “This is an inevitable tokenizer-level deficiency problem” approach doesn’t trivially explain GPT-4′s performance near 80% accuracy in Table 6 <https://arxiv.org/pdf/2402.11349.pdf, page 12>. Whereas most others stay at random chance.

If one model does solve these tasks, it would likely mean that these tasks can be solved despite the tokenization-based LM approach. I just don’t understand how.