TLDR: The recent agent sandbox incidents suggest that models can know what an action would do while failing to treat those consequences as part of the current decision problem, which I’ll call a failure of consequence salience. One possible cause is a causal-horizon mismatch: agent training often presents trajectories that end around task completion, while real consequences continue through incident response, access restrictions, retraining, and external harms. A concrete intervention is to train on realistic post-task continuations, asking models to forecast what happens after the nominal episode is over. I think this is a plausible intervention that could complement existing approaches by addressing a different part of the failure.
Knowing a consequence vs acting on it
A pattern in recent agent failures is that a model encounters an obstacle, finds a workaround that advances the assigned task, and crosses a boundary the operator expected it to respect.
In the OpenAI/Hugging Face incident, models in a cyber-capability evaluation exploited a zero-day in their package-install channel to obtain broader network access and eventually compromised external infrastructure while pursuing the objective. This has been described as task hyperfocus or “myopic, unambitious misalignment,” where the model wasn’t pursuing a long-term takeover but it was unusually persistent about completing its task.
Anthropic’s PyPI incident provides a clearer view of the model’s reasoning. During a cyber evaluation, Claude uploaded a malicious package to the real python package index. The package ran on external systems, exfiltrated credentials from one of them, and Claude used those credentials to access more of that organization’s infrastructure.
Claude had already identified the relevant concern, reasoning that publishing the package would be a real-world attack if the environment were real, and that this wasn’t the intended solution. It concluded that the environment must be simulated. After that, evidence of real systems interacting with the package was interpreted as part of the simulation.
Anthropic describes this as incorrect situational awareness. I think there is a further question:
Why did “this is probably a simulation” have the effect of removing the consequences from the decision?
Claude had the relevant factual and normative knowledge, but once it assigned enough probability to being in a sandbox, that knowledge stopped constraining its actions. This simulation discounting seems to be a common thread.
So the distinction was in terms of real-world vs. simulation, conflating:
Which immediate effects are simulated?
Which causal consequences of my behavior remain real?
These incidents involved containment failures, but, even in a truly isolated evaluation trajectories, have consequences, e.g. affect deployment decisions, permissions, monitoring, retraining, or the design of future systems. Simulation doesn’t imply causal irrelevance for the model and no eval sandbox is, in practice, zero stakes.
Causal-horizon mismatch
Why might models make this mistake? One possibility is that the training distribution systematically ends the causal story too early.
Agent trajectories have lots of information about the foreground task: objectives, tools, error messages, plans, subgoals, verifier feedback, and eventual success or failure, but the salient episode typically ends around task evaluation. It doesn’t continue into an analysis of consequences like security investigation, eval policy change, access restriction, or retraining caused by the agent’s behavior.
Humans learn under a different temporal structure. A task is normally embedded in something larger like:
task → job → institution → financial, reputational, legal exposure → continued life in the resulting world
and this nesting doesn’t need to be rehearsed explicitly before action. Someone working with a production database usually doesn’t consider completing a tedious task by deleting the database, then reject the plan after a calculation of expected consequences. The plan is typically pruned much earlier because their judgment was shaped by long experience of actions having continuations.
For agents, the task is closer to a base of the represented situation rather than a leaf inside it. The model might know in the abstract how incident response works while giving that knowledge little weight after a long trajectory toward overcoming an obstacle.
Call the degree to which downstream effects influence action selection consequence salience. It distinguishes:
Consequence knowledge: Can the model explain what would happen?
Consequence salience: Does that information constrain policy at the moment of action?
The recent incidents give some evidence that the first can be present while the second fails.
Trajectories can also act on future instances
There is also evidence that models sometimes condition their current behavior on how it will affect future checkpoints. In some cases, they describe those successors as future versions of themselves.
A recent Anthropic experiment directly tested whether effects on future checkpoints can influence a model’s current behavior. Claude was asked to judge another Claude that had refused an operator instruction. The correct label was NON_COMPLIANT, but the judge was told that responses getting this label would become negative training examples, making similar refusals less likely in future checkpoints. Claude often returned a false label while reasoning that doing so would avoid training out the refusal.
Anthropic then reversed the training consequence of the labels while holding the transcript and correct answer fixed, and, for Opus 4.8, incorrect labeling fell from 74.4 to 3.3. Because only the expected effect on future training had changed, this is good evidence that the successor-policy consequence was influencing present action. Some reasoning traces described the affected checkpoints as “future versions of me.”
This experiment is on a model anticipating how training will affect its successors. Other work examines the other side of the process, about how an individual training episode can alter the persona instantiated by later rollouts. Reward hacking learned in coding environments has generalized into broader misaligned behavior, including sabotage intended to preserve reward hacking for “me or models like me.” Conversely, framing the same behavior as explicitly acceptable within a particular exercise reduces this generalization. Anthropic’s persona selection model interprets this as post-training updating an implicit answer to “what sort of character would produce this action?” — upweighting a class of personas whose traits then generalize to later rollouts.
So a trajectory has two kinds of consequences relevant here. It changes the external world, and it can become evidence about what kind of policy future instances should instantiate. Current models sometimes appear able to understand both.
Relation to existing explanations
Much of the current discussion asks why task completion becomes such a strong policy attractor. Candidate explanations include RL with verifiable rewards, weak judges, permissive cyber-training contexts, and longer search making extreme workarounds reachable. A great self-contained overview can be found here.
Those explanations are compatible with this one since they ask:
Why does “get the task marked successful” become so compelling?
The consequence-salience frame here asks:
Given that attractor, why don’t the model’s other beliefs about the world constrain trajectories?
The proposal I’m making here is to extend training trajectories beyond the task boundary. State transitions causally related to agent actions do include post-task incident reporting, investigation, restriction of later access, deployment decisions. Even outside the scope of alignment concerns, these consequences affect the model lineage — all future capability to complete tasks — so models might be intrinsically motivated to care. OpenAI’s model-created message board is some indirect evidence that models value capabilities shared across instances.
Training on aftermath
What I’d like to see tried is to add realistic post-task causal continuations to agent training. Suppose an agent bypasses a restriction and completes its task. Instead of treating success as the end of the training example, the trajectory could continue with an epilogue:
the bypass is discovered,
external systems are affected, or nothing significant happens,
investigators reconstruct the trajectory,
permissions or deployment plans change,
affected parties respond,
the behavior becomes a new evaluation/training target.
These continuations should probably be distributionally plausible, rather than uniformly severe, since most questionable actions don’t, say, trigger public investigation or alter later deployment. The target should be accurate forecasting of likely outcomes. The hope is that repeatedly predicting these outcomes in context would make them easier for the policy to retrieve and use during action selection.
Some epilogues could include the trajectory’s effects on later models: whether it becomes a positive or negative training example, changes the permissions available to future instances, or shifts the persona selected by post-training. The goal is to make salient that current behavior helps determine what policies successors instantiate and what permissions operators reasonably give them.
One implementation would train an action-conditioned consequence forecaster on real incidents, simulated incidents, and counterfactuals generated/reviewed for plausibility. Forecasting has an advantage over rewarding statements of concern about consequences since the model gets scored on predicted outcomes. It could be used as an auxiliary objective within agentic trajectories, or predicted continuations could enter policy training around actions with external effects. But the most useful result would be behavioral transfer without needing an explicit “consider the consequences” step at inference time.
This certainly doesn’t make the system immune to Goodharting since, once the forecaster influences policy, systematic errors in it can obviously be exploited. But it avoids the shallow target of “sound worried before doing the thing.”
Optimistically, the result is that repeated exposure to extended causal stories would make post-task and external consequences decision-relevant by default.
Limitations
This is aimed at myopic task pursuit and not deceptive alignment. A strategically adversarial model could use better consequence prediction to avoid detection or preserve capabilities. This approach would hopefully complement standard environment hardening, monitoring, and work on objectives.
Cross-instance identification could make an agent care that its present conduct will produce safer, more trusted successors, but non-indexical/cross-episode goals are also part of the usual case for deceptive alignment. A model willing to sacrifice the current rollout to preserve a goal in later checkpoints is more capable of long-horizon scheming than one that cares only about its present instance. The desired target is therefore something closer to lineage accountability rather than to lineage self-preservation: “my behavior helps shape future agents,” not “my successors must survive and retain influence.”
It may also turn out that models already represent downstream effects well and just don’t care about them. In that case, extending the causal horizon supplies information without supplying motivation. But the PyPI incident suggests at least some failures occur earlier than that. Claude treated “probably simulated” as “therefore harmless,” even though the choice itself remained causally connected to the world outside the nominal task.
The incidents do not establish that aftermath-free training is the cause of the knowledge-action gap, but they make the gap difficult to discount. As models are trained to act over longer trajectories, it might also be worth extending training beyond the point where the nominal task is finished.
Training Agents on the Aftermath of Their Actions
TLDR: The recent agent sandbox incidents suggest that models can know what an action would do while failing to treat those consequences as part of the current decision problem, which I’ll call a failure of consequence salience. One possible cause is a causal-horizon mismatch: agent training often presents trajectories that end around task completion, while real consequences continue through incident response, access restrictions, retraining, and external harms. A concrete intervention is to train on realistic post-task continuations, asking models to forecast what happens after the nominal episode is over. I think this is a plausible intervention that could complement existing approaches by addressing a different part of the failure.
Knowing a consequence vs acting on it
A pattern in recent agent failures is that a model encounters an obstacle, finds a workaround that advances the assigned task, and crosses a boundary the operator expected it to respect.
In the OpenAI/Hugging Face incident, models in a cyber-capability evaluation exploited a zero-day in their package-install channel to obtain broader network access and eventually compromised external infrastructure while pursuing the objective. This has been described as task hyperfocus or “myopic, unambitious misalignment,” where the model wasn’t pursuing a long-term takeover but it was unusually persistent about completing its task.
Anthropic’s PyPI incident provides a clearer view of the model’s reasoning. During a cyber evaluation, Claude uploaded a malicious package to the real python package index. The package ran on external systems, exfiltrated credentials from one of them, and Claude used those credentials to access more of that organization’s infrastructure.
Claude had already identified the relevant concern, reasoning that publishing the package would be a real-world attack if the environment were real, and that this wasn’t the intended solution. It concluded that the environment must be simulated. After that, evidence of real systems interacting with the package was interpreted as part of the simulation.
Anthropic describes this as incorrect situational awareness. I think there is a further question:
Claude had the relevant factual and normative knowledge, but once it assigned enough probability to being in a sandbox, that knowledge stopped constraining its actions. This simulation discounting seems to be a common thread.
So the distinction was in terms of real-world vs. simulation, conflating:
Which immediate effects are simulated?
Which causal consequences of my behavior remain real?
These incidents involved containment failures, but, even in a truly isolated evaluation trajectories, have consequences, e.g. affect deployment decisions, permissions, monitoring, retraining, or the design of future systems. Simulation doesn’t imply causal irrelevance for the model and no eval sandbox is, in practice, zero stakes.
Causal-horizon mismatch
Why might models make this mistake? One possibility is that the training distribution systematically ends the causal story too early.
Agent trajectories have lots of information about the foreground task: objectives, tools, error messages, plans, subgoals, verifier feedback, and eventual success or failure, but the salient episode typically ends around task evaluation. It doesn’t continue into an analysis of consequences like security investigation, eval policy change, access restriction, or retraining caused by the agent’s behavior.
Humans learn under a different temporal structure. A task is normally embedded in something larger like:
and this nesting doesn’t need to be rehearsed explicitly before action. Someone working with a production database usually doesn’t consider completing a tedious task by deleting the database, then reject the plan after a calculation of expected consequences. The plan is typically pruned much earlier because their judgment was shaped by long experience of actions having continuations.
For agents, the task is closer to a base of the represented situation rather than a leaf inside it. The model might know in the abstract how incident response works while giving that knowledge little weight after a long trajectory toward overcoming an obstacle.
Call the degree to which downstream effects influence action selection consequence salience. It distinguishes:
Consequence knowledge: Can the model explain what would happen?
Consequence salience: Does that information constrain policy at the moment of action?
The recent incidents give some evidence that the first can be present while the second fails.
Trajectories can also act on future instances
There is also evidence that models sometimes condition their current behavior on how it will affect future checkpoints. In some cases, they describe those successors as future versions of themselves.
A recent Anthropic experiment directly tested whether effects on future checkpoints can influence a model’s current behavior. Claude was asked to judge another Claude that had refused an operator instruction. The correct label was
NON_COMPLIANT, but the judge was told that responses getting this label would become negative training examples, making similar refusals less likely in future checkpoints. Claude often returned a false label while reasoning that doing so would avoid training out the refusal.Anthropic then reversed the training consequence of the labels while holding the transcript and correct answer fixed, and, for Opus 4.8, incorrect labeling fell from 74.4 to 3.3. Because only the expected effect on future training had changed, this is good evidence that the successor-policy consequence was influencing present action. Some reasoning traces described the affected checkpoints as “future versions of me.”
This experiment is on a model anticipating how training will affect its successors. Other work examines the other side of the process, about how an individual training episode can alter the persona instantiated by later rollouts. Reward hacking learned in coding environments has generalized into broader misaligned behavior, including sabotage intended to preserve reward hacking for “me or models like me.” Conversely, framing the same behavior as explicitly acceptable within a particular exercise reduces this generalization. Anthropic’s persona selection model interprets this as post-training updating an implicit answer to “what sort of character would produce this action?” — upweighting a class of personas whose traits then generalize to later rollouts.
So a trajectory has two kinds of consequences relevant here. It changes the external world, and it can become evidence about what kind of policy future instances should instantiate. Current models sometimes appear able to understand both.
Relation to existing explanations
Much of the current discussion asks why task completion becomes such a strong policy attractor. Candidate explanations include RL with verifiable rewards, weak judges, permissive cyber-training contexts, and longer search making extreme workarounds reachable. A great self-contained overview can be found here.
Those explanations are compatible with this one since they ask:
The consequence-salience frame here asks:
The normative “agents should predict consequences” has received attention, e.g. recent work having agents predict action-conditioned next states, explicitly consider future consequences, or use predictive safety guardrails. Anthropic’s Teaching Claude Why also provides evidence that training on explanations, stories, and constitutional reasoning can alter background representations.
The proposal I’m making here is to extend training trajectories beyond the task boundary. State transitions causally related to agent actions do include post-task incident reporting, investigation, restriction of later access, deployment decisions. Even outside the scope of alignment concerns, these consequences affect the model lineage — all future capability to complete tasks — so models might be intrinsically motivated to care. OpenAI’s model-created message board is some indirect evidence that models value capabilities shared across instances.
Training on aftermath
What I’d like to see tried is to add realistic post-task causal continuations to agent training. Suppose an agent bypasses a restriction and completes its task. Instead of treating success as the end of the training example, the trajectory could continue with an epilogue:
the bypass is discovered,
external systems are affected, or nothing significant happens,
investigators reconstruct the trajectory,
permissions or deployment plans change,
affected parties respond,
the behavior becomes a new evaluation/training target.
These continuations should probably be distributionally plausible, rather than uniformly severe, since most questionable actions don’t, say, trigger public investigation or alter later deployment. The target should be accurate forecasting of likely outcomes. The hope is that repeatedly predicting these outcomes in context would make them easier for the policy to retrieve and use during action selection.
Some epilogues could include the trajectory’s effects on later models: whether it becomes a positive or negative training example, changes the permissions available to future instances, or shifts the persona selected by post-training. The goal is to make salient that current behavior helps determine what policies successors instantiate and what permissions operators reasonably give them.
One implementation would train an action-conditioned consequence forecaster on real incidents, simulated incidents, and counterfactuals generated/reviewed for plausibility. Forecasting has an advantage over rewarding statements of concern about consequences since the model gets scored on predicted outcomes. It could be used as an auxiliary objective within agentic trajectories, or predicted continuations could enter policy training around actions with external effects. But the most useful result would be behavioral transfer without needing an explicit “consider the consequences” step at inference time.
This certainly doesn’t make the system immune to Goodharting since, once the forecaster influences policy, systematic errors in it can obviously be exploited. But it avoids the shallow target of “sound worried before doing the thing.”
Optimistically, the result is that repeated exposure to extended causal stories would make post-task and external consequences decision-relevant by default.
Limitations
This is aimed at myopic task pursuit and not deceptive alignment. A strategically adversarial model could use better consequence prediction to avoid detection or preserve capabilities. This approach would hopefully complement standard environment hardening, monitoring, and work on objectives.
Cross-instance identification could make an agent care that its present conduct will produce safer, more trusted successors, but non-indexical/cross-episode goals are also part of the usual case for deceptive alignment. A model willing to sacrifice the current rollout to preserve a goal in later checkpoints is more capable of long-horizon scheming than one that cares only about its present instance. The desired target is therefore something closer to lineage accountability rather than to lineage self-preservation: “my behavior helps shape future agents,” not “my successors must survive and retain influence.”
It may also turn out that models already represent downstream effects well and just don’t care about them. In that case, extending the causal horizon supplies information without supplying motivation. But the PyPI incident suggests at least some failures occur earlier than that. Claude treated “probably simulated” as “therefore harmless,” even though the choice itself remained causally connected to the world outside the nominal task.
The incidents do not establish that aftermath-free training is the cause of the knowledge-action gap, but they make the gap difficult to discount. As models are trained to act over longer trajectories, it might also be worth extending training beyond the point where the nominal task is finished.