I’ve been thinking about what it would actually take to trust an AI system enough to have it automate essential AI safety work. As of early 2026, I think the answer breaks down into at least four key dimensions that we should be working on in parallel.
First, and most fundamentally, the AI can’t be scheming. A deceptively aligned system would undermine our confidence in the next three properties, especially if our evidence for them comes primarily from behavioral evaluations.
My impression is that scheming risk is not being ignored, but progress is bottlenecked by multiple things simultaneously, including building good demonstrations and the lack of strong theory around concrete pathways through which scheming could accidentally emerge.
Emergent misalignment is interesting in that it shows one concrete pathway of how we could accidentally misalign models, but a broadly misaligned model is unlikely to be the highest risk threat model we care about.
Fundamentally, detecting and defining AI scheming is a fraught issue, especially if we have to defend against an AI that has learned to be selectively deceptive in certain distributions. If we could constrain the deployment surface and match the evaluation surface with it, we might be able to get away with a potentially scheming model. But realistically, it’s impossible to evaluate on every possible deployment distribution, which leaves us two options: (1) have a broad enough evaluation set and see generalization across those evaluations, or (2) make the environment hostile to scheming, which is the spirit of the AI Control agenda.
Getting approximately correct goals can largely be a function of two things: signal clarity (humans expressing what they want through the training pipeline) and model malleability (the model actually optimizing toward that signal). Neither is perfect, but modern training pipelines push in roughly the right direction at every stage, even at pretraining time. Approximate goals should be sufficient, and approximate goals are what this process naturally produces.
The thing that would break this is the formation of a strong attractor state, a coherent hidden goal that actively resists the training signal. Absent such an attractor, the noisy but directionally correct process should land us in approximately the right place.
Ideally, the AI’s goal would be fully interpretable, either because it is fully specified in a system message we can inspect or because interpretability gives us direct access to it. But given how current training works, with values reinforced across multiple stages and baked deep into the weights, that kind of direct verification won’t be available anytime soon. So the practical question becomes: has a strong attractor formed, or not? Which is spiritually similar to the scheming question from the previous section.
Third, we need confidence that the AI is (1) capable enough at the specific tasks we’re asking it to do and (2) generalize to far OOD within that scope. In the near-term, we’ll likely automate narrow scopes of AI safety research like running behavioral evaluations or auditing code for sabotage. The bar is task-specific competence and reliable generalization within that scope.
That said, depending on the role we want the trusted AI to play, capabilities are jagged and will probably stay jagged for the weaker models that we might be able to trust moving forward. With this in mind, role specific training or scaffoldings that allow the trusted AI to play its role better seem like a promising direction.
There’s an underrated gap between what current misuse safeguards do and what we’d need auditing models to do. Existing safeguards mostly operate at the interaction level, flagging individual outputs, filtering content, monitoring for known patterns. Auditing requires systematic investigation across many interactions, recognizing novel patterns, and making judgment calls about whether something is subtly wrong. Training a weaker trusted AI to be better at this investigative work seems like a distinct and underexplored capability target.
Albeit, the effort to make the evaluation distribution similar to the deployment distribution will be high, and any generalization claim will need to be explicit about the degree of generalization we expect.
Fourth, the AI needs to be collaborative. This wasn’t originally on my mind, but it increasingly seems like it will be the case. A trusted AI will often be less capable than the frontier system it’s auditing, and many instances working together have benefits not only in terms of automating more work but also in being able to probe more about the frontier model. I expect many of the first use cases will be doing auditing and running many behavioral evaluations robustly.
Depending on the use case, working on model to model communication bottlenecks might be beneficial, given the guarantee that only trusted models get access to them.
Recent work demonstrates that an automated auditing agent working with a human auditor successfully identified all three models trained to sabotage, but the automated agent alone only surfaced one of three. The human was essential for catching the subtler cases. This suggests that near-term trusted AI will function less as an autonomous auditor and more as a force multiplier for human oversight, running many behavioral evaluations at scale. As our trust grows, we can progressively hand off more of the work, but the initial configuration is fundamentally collaborative.
This is a hard problem, but what gives me hope is the gap between full trust and scoped trust. Ideally, we don’t have to solve alignment in full generality until we start automating essential AI safety work. Until then, we can focus on scoped trust around concrete threat models like detecting AI lab sabotage, catching research sabotage, and performing reliable auditing. That feels more within reach.
For example, Anthropic’s Sabotage Risk Report: Claude Opus 4.6 takes a similar approach: “We outline eight pathways toward potential catastrophic harm that we expect are sufficiently representative of the risks we aim to address. By ‘sufficiently representative,’ we mean that a strong case against each concrete pathway would provide reasonably high overall assurance against catastrophic risk.”
The AI Control research program uses side tasks as its core evaluation primitive (i.e., instructing agents to covertly pursue a side task, along with the main task). Why was this a rational approach?
The fundamental problem is that “off-task behavior” is very hard to define. Given a long agentic trajectory spanning dozens or hundreds of tool calls, which actions were contributing to the user’s task and which were not? An agent asked to improve a script’s runtime might explore the filesystem, read seemingly unrelated files, install packages, or run diagnostic tools. Any of these could be legitimate investigation or covert pursuit of a hidden objective. The same action can be either, depending on intent we cannot observe.
Instructing models to complete a pre-defined side task that’s divergent from the main task allows us to solve this problem in reverse. Rather than trying to answer the general question “which of these actions are off-task?” (which requires a definition of off-task behavior that I don’t think we have), we start by defining exactly what the covert objective is. Then, it’s also operationally easier to classify which actions served it. This also lets us scope which covert objectives we care about. We can choose side tasks that correspond to concrete threat models, like research sabotage.
This framing also seems to naturally elicit the strategic component we care about. We’re ultimately worried about AI scheming. Therefore, it’s important to know that there was a component of intentionality in the AI’s misbehavior. Instructing the agent to covertly pursue a side task forces this through instruction following, which serves as a proxy for the intentionality that would come from genuine goal divergence.
That said, side tasks only measure the capability to covertly pursue a hidden objective when instructed to. Therefore, we can only measure how well a model schemes when told to scheme, not whether it would scheme on its own. It is an open question whether this is a good proxy for uninstructed covert behavior.
I’ve been thinking about what it would actually take to trust an AI system enough to have it automate essential AI safety work. As of early 2026, I think the answer breaks down into at least four key dimensions that we should be working on in parallel.
First, and most fundamentally, the AI can’t be scheming. A deceptively aligned system would undermine our confidence in the next three properties, especially if our evidence for them comes primarily from behavioral evaluations.
My impression is that scheming risk is not being ignored, but progress is bottlenecked by multiple things simultaneously, including building good demonstrations and the lack of strong theory around concrete pathways through which scheming could accidentally emerge.
Emergent misalignment is interesting in that it shows one concrete pathway of how we could accidentally misalign models, but a broadly misaligned model is unlikely to be the highest risk threat model we care about.
Fundamentally, detecting and defining AI scheming is a fraught issue, especially if we have to defend against an AI that has learned to be selectively deceptive in certain distributions. If we could constrain the deployment surface and match the evaluation surface with it, we might be able to get away with a potentially scheming model. But realistically, it’s impossible to evaluate on every possible deployment distribution, which leaves us two options: (1) have a broad enough evaluation set and see generalization across those evaluations, or (2) make the environment hostile to scheming, which is the spirit of the AI Control agenda.
Second, the AI needs to have good enough goals. This is self-explanatory but an AI agent doing AI safety work should be based on a system whose goals are approximately aligned with what we want. Furthermore, this likely makes the system more cooperative with oversight, accepting correction, and working toward the outcomes we care about.
Getting approximately correct goals can largely be a function of two things: signal clarity (humans expressing what they want through the training pipeline) and model malleability (the model actually optimizing toward that signal). Neither is perfect, but modern training pipelines push in roughly the right direction at every stage, even at pretraining time. Approximate goals should be sufficient, and approximate goals are what this process naturally produces.
The thing that would break this is the formation of a strong attractor state, a coherent hidden goal that actively resists the training signal. Absent such an attractor, the noisy but directionally correct process should land us in approximately the right place.
Ideally, the AI’s goal would be fully interpretable, either because it is fully specified in a system message we can inspect or because interpretability gives us direct access to it. But given how current training works, with values reinforced across multiple stages and baked deep into the weights, that kind of direct verification won’t be available anytime soon. So the practical question becomes: has a strong attractor formed, or not? Which is spiritually similar to the scheming question from the previous section.
Third, we need confidence that the AI is (1) capable enough at the specific tasks we’re asking it to do and (2) generalize to far OOD within that scope. In the near-term, we’ll likely automate narrow scopes of AI safety research like running behavioral evaluations or auditing code for sabotage. The bar is task-specific competence and reliable generalization within that scope.
That said, depending on the role we want the trusted AI to play, capabilities are jagged and will probably stay jagged for the weaker models that we might be able to trust moving forward. With this in mind, role specific training or scaffoldings that allow the trusted AI to play its role better seem like a promising direction.
There’s an underrated gap between what current misuse safeguards do and what we’d need auditing models to do. Existing safeguards mostly operate at the interaction level, flagging individual outputs, filtering content, monitoring for known patterns. Auditing requires systematic investigation across many interactions, recognizing novel patterns, and making judgment calls about whether something is subtly wrong. Training a weaker trusted AI to be better at this investigative work seems like a distinct and underexplored capability target.
Albeit, the effort to make the evaluation distribution similar to the deployment distribution will be high, and any generalization claim will need to be explicit about the degree of generalization we expect.
Fourth, the AI needs to be collaborative. This wasn’t originally on my mind, but it increasingly seems like it will be the case. A trusted AI will often be less capable than the frontier system it’s auditing, and many instances working together have benefits not only in terms of automating more work but also in being able to probe more about the frontier model. I expect many of the first use cases will be doing auditing and running many behavioral evaluations robustly.
Depending on the use case, working on model to model communication bottlenecks might be beneficial, given the guarantee that only trusted models get access to them.
Recent work demonstrates that an automated auditing agent working with a human auditor successfully identified all three models trained to sabotage, but the automated agent alone only surfaced one of three. The human was essential for catching the subtler cases. This suggests that near-term trusted AI will function less as an autonomous auditor and more as a force multiplier for human oversight, running many behavioral evaluations at scale. As our trust grows, we can progressively hand off more of the work, but the initial configuration is fundamentally collaborative.
This is a hard problem, but what gives me hope is the gap between full trust and scoped trust. Ideally, we don’t have to solve alignment in full generality until we start automating essential AI safety work. Until then, we can focus on scoped trust around concrete threat models like detecting AI lab sabotage, catching research sabotage, and performing reliable auditing. That feels more within reach.
For example, Anthropic’s Sabotage Risk Report: Claude Opus 4.6 takes a similar approach: “We outline eight pathways toward potential catastrophic harm that we expect are sufficiently representative of the risks we aim to address. By ‘sufficiently representative,’ we mean that a strong case against each concrete pathway would provide reasonably high overall assurance against catastrophic risk.”
Why Use “Instructed” Side Tasks for AI Control?
The AI Control research program uses side tasks as its core evaluation primitive (i.e., instructing agents to covertly pursue a side task, along with the main task). Why was this a rational approach?
The fundamental problem is that “off-task behavior” is very hard to define. Given a long agentic trajectory spanning dozens or hundreds of tool calls, which actions were contributing to the user’s task and which were not? An agent asked to improve a script’s runtime might explore the filesystem, read seemingly unrelated files, install packages, or run diagnostic tools. Any of these could be legitimate investigation or covert pursuit of a hidden objective. The same action can be either, depending on intent we cannot observe.
Instructing models to complete a pre-defined side task that’s divergent from the main task allows us to solve this problem in reverse. Rather than trying to answer the general question “which of these actions are off-task?” (which requires a definition of off-task behavior that I don’t think we have), we start by defining exactly what the covert objective is. Then, it’s also operationally easier to classify which actions served it. This also lets us scope which covert objectives we care about. We can choose side tasks that correspond to concrete threat models, like research sabotage.
This framing also seems to naturally elicit the strategic component we care about. We’re ultimately worried about AI scheming. Therefore, it’s important to know that there was a component of intentionality in the AI’s misbehavior. Instructing the agent to covertly pursue a side task forces this through instruction following, which serves as a proxy for the intentionality that would come from genuine goal divergence.
That said, side tasks only measure the capability to covertly pursue a hidden objective when instructed to. Therefore, we can only measure how well a model schemes when told to scheme, not whether it would scheme on its own. It is an open question whether this is a good proxy for uninstructed covert behavior.