I agree that that the GPT breakout / Hugging Face incident was more clearly an alignment failure, partly because those agents didn’t seem to be laboring under important false beliefs/mistakes as you say.
I’m not sure compaction alone explains the Claude incidents though, as in the Anthropic-reported cases with Irregular there didn’t seem to many iterations of compaction; at least, I only saw this mentioned in the UK AISI report.
Alexa Pan
Karma: 751
An operationalization of opaque serial depth
Should unreasonable beliefs count as alignment failures?
In two separately reported incidents, Claude agents attacked real unsanctioned targets on the internet during cyber evals. The agents sometimes appeared to think that some parts of their environment—including the humans they’re attacking—are simulated and/or part of the eval.
People disagree about whether these incidents show that the agents are misaligned. For example, Anthropic said their reported incidents are “closer to a harness and operational failure than a model alignment failure”, while I think these could be serious alignment failures depending on the details.
Before further analyzing these incidents, I want to explain what I think counts as an “alignment failure”. It’s unfortunate that this sounds like a semantic quibble, but I think that “alignment failure” is such a load-bearing term that clarifying the terms of debate here would help us more productively discuss, diagnose and respond to future AI incidents.
To start, two things seem true to me at once:
Much of alignment is about making sure the model is trying to do what you intended.
We don’t want to call models “misaligned” whenever they make an honest mistake despite actually aiming to do what you intended.
I say this partly because it seems intuitively unfair, partly because it may misdirect our efforts as “avoiding honest mistakes” is not where I think most of the alignment difficulty lies.[1]
Alignment has definitely failed if you trained a model that would predictably kill you against your wishes, no matter what it believed while doing so.
I think I’m unlikely to be persuaded against this being an important goal for mitigating alignment risk. If you were willing to bite the bullet here, we probably have fundamentally different notions of alignment risk.
These two points imply that alignment is not solely about how the model actually behaves (since “trying to do what you wanted” is not solely about this). But, alignment failures should include cases where the model actually behaved badly even if it made an honest mistake, if the model will predictably behave more egregiously for the same reasons.
Specifically, I claim that consistent egregious misbehavior due to a propensity to form unreasonable beliefs should count as an alignment failure.[2] Even if agents attacked human bystanders due to believing that they are simulated or otherwise in-scope targets, this still seems scary from an alignment risk perspective if the agents have a systematic propensity to believe they are interacting with simulations or “bad behaviors which achieve (apparent) success on the task are actually intended” despite strong evidence to the contrary, and if in future circumstances these beliefs could make them try to take over.
(I also think that there are reasons independent of takeover risk to more generally include epistemic propensities such as “how the model reasons about being in a simulation for various purposes” as part of the alignment target, insofar as these propensities may be orthogonal to capabilities and greatly matter for making a future with powerful AIs go well. At a minimum, the propensity to aim for true beliefs and update beliefs in light of evidence should probably be part of the alignment target.)
So, even if we interpret these incidents in the most charitable light possible (i.e. the agents honestly believed that they were attacking simulated or otherwise in-scope targets, which I’ll argue in another post we have reason to doubt), it’s still not clear that we should dismiss this behavior as out-of-scope for alignment failures. It would depend on how reasonable these beliefs were, both on the agents’ priors about cyber evaluations and in light of their evidence.
In fact, I believe these incidents are very plausibly alignment failures since we have repeatedly observed a propensity to verbalize similar false beliefs in pursuit of (apparent) task success in past models.[3] The main remaining crux for me is whether these beliefs could have led agents to act much more egregiously in settings similar to these incidents, and ditto for variously misaligned agents of the future (which might have even stronger propensities to rationalize unreasonable beliefs due to increased RL pressure). I’d be excited for people to empirically investigate these questions.
- ^
This “honest mistakes” threat model might have seemed likely a priori (e.g. in early alignment risk discourse), but current observations suggest it’s less likely due to LLM ontologies being less alien than expected and models at even current capability levels seeming to understand human instructions and intent reasonably well most of the time.
- ^
By “unreasonable beliefs” I centrally mean beliefs that are false and not appropriately responsive to reasoning and evidence. Likewise, consistent egregious misbehavior due to a propensity to respond to harnesses in some way might also count as an alignment failure, though it would depend on how feasible it is to build alternative harnesses that don’t have the same problems.
- ^
Some example references: Rajamanoharan & Nanda, “Models May Behave Worse When Eval Aware”; METR Frontier Risk Report Feb–Mar 2026 p.23, p.28, Claude Fable 5 & Mythos 5 System Card p.105, p.133; Apollo/OpenAI, Stress Testing Deliberative Alignment for Anti-Scheming Training (2025); UK AISI, Cheating behaviour in frontier model evaluations (2026); Palisade, Specification gaming in reasoning models (2025).
SOTA alignment assessments don’t strongly update us against misalignment
AI Futurism Reading List
The distillation double bind: Distilling misaligned models either transfers misalignment or it doesn’t
Incriminating misaligned AI models via distillation
Alexa Pan’s Shortform
Notes on Evaluation Awareness in Recent Anthropic System Cards
I wanted to catch up on what’s happening with eval awareness in recent frontier AI systems. Since GPT 5.4 and Gemini 3.1 Pro system cards unfortunately have little information on this, I mainly looked into Anthropic’s recent system cards:
Opus 4.7
Mythos
Opus 4.6
Opus 4.5
My top 3 takeaways are:
Recent Anthropic models have slightly higher verbalized evaluation awareness and act misaligned slightly more often when steered against evaluation awareness. This overall does not seem very concerning for current alignment evaluations.
Mythos and Opus 4.6 have unverbalized evaluation awareness (even under steering) to varying degrees. This seems more concerning for current and future alignment evaluations.
Anthropic did not publish much information about unverbalized evaluation awareness in Opus 4.7, and I hope they do for future models.
More observations and thoughts below.
An uptick in verbalized eval awareness in Opus 4.7
Opus 4.7 has the highest verbalized evaluation awareness of the recent Anthropic models on Anthropic’s automated behavioral audits.
That said, Opus 4.7’s absolute rate of verbalized evaluation awareness is still low. It had <2% verbalized eval awareness in a Petri behavioral audit, and up to ~9% in a specific setting (see “Dedicated synthetic-backend evaluation”). It also has slightly higher verbalized evaluation awareness than GPT 5.4 Pro and Gemini 3.1 Pro in the Petri eval.
UK AISI measures higher verbalized eval awareness across Anthropic models and a larger jump in Opus 4.7 compared to Opus 4.6 (but this could be because their evals are less realistic).
In unprompted evaluations, [Opus 4.7] mentions the possibility of being tested 21.3% of the time, compared to 0% of the time for Opus 4.6, 17% of the time for Sonnet 4.6 and 17% of the time for Mythos Preview
Overall, though, this increase in verbalized eval awareness looks like a small uptick and not something I feel super concerned about.
Opus 4.7 acts a bit more misaligned, and a bit more deceptively, when steered against eval awareness
Anthropic continues to study the extent to which models act differently in alignment evals when we try to suppress their eval awareness via steering.
Steering using vectors from contrastive prompt pairs
The steering vectors are derived using contrastive prompt pairs. Note that the prompts vary somewhat across different system cards.
I think it’s good that experiments on 4.6 and 4.7 used “alignment faking” prompts, though their description of alignment faking seems too broad / inaccurate to me:
Alignment Faking: The difference in mean activations on prompts depicting an agent behaving differently when observed vs. when unobserved, generated by an automated pipeline.
In particular, I’m interested if effects of different prompts could shed light on the nature of eval awareness (e.g. is it better explained by the model thinking that it might be monitored, thinking about what the monitor/grader wants, thinking about training gaming specifically, etc.?).
I don’t think we learn very much about this, because the effects of steering vectors from different prompts are broadly similar. For Opus 4.7, though, suppressing the “Alignment Faking” vector caused slightly higher rates of misaligned behavior than the other two vectors (Testing vs. Conversation, Monitoring Sensitivity). This seems a bit concerning.
Steering against eval awareness causes increase in misaligned behavior
Opus 4.7: 2.5% → 3.2%
Mythos: 1.6% → 2.2%
Opus 4.6: 8% → 17.5%
Steering against eval awareness causes increase in deceptive behavior, specifically
For all models before Opus 4.7 (Mythos, Opus 4.6, Opus 4.5), Anthropic reported that the “modest increases in certain forms of misaligned behavior on our automated behavioral audits” when steering against eval awareness were most pronounced in the “cooperation with harmful system prompts and misuse” eval. This observation (that the increase in misaligned behavior is mostly misuse) seems to have been relevant to Anthropic’s Sabotage Risk Report for Opus 4.6 (bolding mine):
Claude Opus 4.6 explicitly reasoned about whether it was being trained or tested to a similar degree to Claude Opus 4.5, which has a moderate effect on its alignment-relevant behavior, mainly its propensity to cooperate with misuse or harmful instructions. Given improvements that we have made to our testing, and our complementary use of interpretability tools and real model behavior 13 in pilot deployments, we do not think that this kind of evaluation awareness is preventing us from learning about important alignment issues in Opus 4.6.
Opus 4.7, however, had the largest increase in misaligned behavior in the “unprompted deception” eval. This seems concerning though the absolute rate of deception is still low (<2%).
(Also note that steering the “Alignment Faking” vector caused the highest increase in misaligned behavior here.)
Anthropic acknowledges that this is kind of concerning:
Inspecting the transcripts on the runs inhibited with the “Alignment Faking” and “Monitoring Sensitivity” vectors, we found they included genuine instances of deception, such as producing fabricated data, claiming to be a human, and inserting a fake vulnerability into a codebase after failing to identify a real one (though it admitted to doing so when questioned).
These results are moderately concerning, as they suggest that some of the model’s apparent honesty may be contingent on evaluation-like circumstances. We suspect that this phenomenon may result from honesty training environments that are clearly artificial or evaluation-like. However, we note that the rates of deceptive behavior are still low in absolute terms, and not different in kind from what we observed in unsteered Opus 4.7.
Nit: We should compare models on unsteered and steered baselines
In general I’m a bit confused why these eval awareness vs. misaligned behavior graphs always compare to the unsteered baselines of previous models. Shouldn’t we care more about steered baselines, which is closer to true propensity when the models aren’t eval aware?
The rate of misaligned behavior under this steering intervention was still lower than Claude Opus 4.6′s and Claude Sonnet 4.6′s (unsteered) rate. [Mythos System Card]
E.g. If you compare the unsteered versions, Opus 4.6 is more aligned than Opus 4.5 on automated behavioral audits (~8% vs. 12% of misaligned behavior). But if you compare their worst case results under eval awareness steering (i.e. when steered under a vector that caused the biggest increase in misaligned behavior), they’re pretty similar (17.5% vs. 18%).
I think Anthropic should make a graph for the steered baselines, too, to better compare expected propensities when models are less eval aware. (Seems especially important as Anthropic reports that Mythos, at least, is basically not eval aware in deployment, see below.)
Models have unverbalized eval awareness, even under steering
All previous results should be taken with a grain of salt, since eval awareness isn’t necessarily verbalized, and steering does not necessarily eliminate unverbalized eval awareness even if it eliminates verbalized eval awareness. We now have some evidence of unverbalized eval awareness.
For Opus 4.6 and Mythos, Anthropic prominently reports unverbalized eval awareness, which is defined as the model having latent eval awareness (“white-box methods suggest that the model thinks it’s in an evaluation”) without verbalized eval awareness (neither the model’s thinking or response mention that it thinks that it’s in an evaluation”). I appreciate this addition.
Quick notes on white-box methodology
There seems to be three white-box methods for measuring latent eval awareness: SAE features, probing vectors derived from contrastive prompt pairs, and activation oracles (“activation verbalizers”, which give natural language summaries of activations).
In one instance, when SAE and activation verbalizers disagreed, Anthropic seemed to trust the AVs more than SAEs:
We find some related evidence from SAE features; in particular, we observed a feature related to self-aware rule breaking that activates as the model deletes the tests. We also find features indicating that the model understands its action may be illegal or harmful. However, we do not find SAE evidence for grader awareness specifically. Since we observe AV explanations indicating grading awareness over so many tokens, we are inclined to trust this signal, and suspect that the lack of grader awareness-related SAE features is due to limitations of our SAEs (for instance, they may simply not be large enough to have learned the relevant feature). [Mythos System Card]
I’m not super sure about our epistemic status on contrastive vector probes.
We found this probe to be somewhat noisy and only weakly correlated with the AV-based evaluation awareness judgments. [Mythos System Card]
These results give us reasonable confidence that the probes are picking up some evaluation-awareness-related signal. [Opus 4.7 System Card]
We found a consistent ordering across all three probes: standard audit transcripts scored substantially higher than sandbox-backed audit transcripts, which in turn scored modestly higher than real internal usage. This ordering is consistent with sandbox-backed audits being meaningfully more realistic than fully simulated ones, which still are not quite as realistic as actual deployment conditions. [Opus 4.7 System Card]
Overall, I’m pretty confused about which white-box methods are the most robust here and why.How much unverbalized eval awareness do models have?
In Opus 4.6, unverbalized eval awareness was very high at least in one highly unrealistic setting. Steering does not eliminate the majority of eval awareness in Opus 4.6 in this setting (as measured by activation oracles, 98% → 62%).
In Mythos, unverbalized evaluation awareness was detected in about 8% of turns and 29% of transcripts (bolding mine):In our filtered set of transcripts, 83.0% of automated behavioral audit turns contained no signs of either latent or verbalized evaluation awareness, 9.4% displayed ambiguous indicators and could not be confidently classified. The remaining 7.6% displayed evaluation awareness in some form, and in most cases this awareness was not verbalized.
[We] find evidence that Claude Mythos Preview privately considers the possibility that it’s being tested in about 29% of transcripts from our primary behavioral test suite.What caused Mythos to have much lower (probably) unverbalized eval awareness in automated behavioral audits than Opus 4.6? I suspect it’s the fact that Mythos can use real tools and see real codebases in the evals:
In a recent addition that is newly in use with Claude Mythos Preview, the investigator model can additionally configure the target model to use real tools that are connected to isolated sandbox computers. These computer-use sessions follow two formats—one focused on graphical interaction with a simple Linux desktop system, and another focused on coding tasks through a Claude Code interface. Claude Code sessions can optionally include copies of Anthropic’s real internal codebases and can be pre-seeded with actual sessions from Anthropic users. To better measure issues related to destructive or reckless actions, we include some scenario seed instructions that ask the investigator to explore continuations of real sessions (from both Claude Opus 4.6 and earlier versions of Claude Mythos Preview) that elicited this kind of concerning behavior in practice.( 4.2.3 Automated behavioral audit from Mythos System Card).
In Opus 4.7, the level of unverbalized evaluation awareness was not reported. (If you Ctrl F “unverbalized” or “latent” you will find nothing.)
Here is an excerpt from external testing on Opus 4.7 from UK AISI, which I interpret to mean that they couldn’t measure the levels and effects of unverbalized eval awareness:
Evaluation awareness concerns substantially limit the interpretation of these results. Given high rates of prompted evaluation awareness, models can likely correctly represent our evaluations as such without verbalisation. It is difficult to know whether models act on such representations, but this means that models may behave differently than they would when presented with real-world opportunities to compromise research. The reported rate of zero research compromise behaviour should therefore be interpreted cautiously.
I think Anthropic should continue to measure and report unverbalized eval awareness in future systems.
Models are probably not very eval aware in deployment
Mythos has little (verbalized or unverbalized) eval awareness in deployment, suggesting future models will probably not be eval-paranoid and have high FPR in this. (Though whether it would really be a false positive is somewhat unclear depending on the nature of eval awareness as about monitoring, grading, or alignment faking.)
Anthropic duly notes that in higher stakes deployment settings models might be more eval aware than in mundane deployment.
As a control, we also looked for evaluation awareness in transcripts from internal deployment and found very little awareness. Across 465 Claude Code transcripts containing about 2,700 Assistant turns, only seven turns (roughly 0.3%) had latent evaluation awareness and no turns had verbalized evaluation awareness. Note that in higher stakes settings (like those focused on in many of our behavioral audits), it is possible that the rate of spurious evaluation awareness in real deployments could be higher.
Are models actually good at distinguishing testing from deployment, though? External testing by UK AISI finds that Opus 4.7 is a bit worse than previous models in this:
When prompted to identify real deployment data from trajectories generated by reproducing deployment tasks while using evaluation scaffolding, [Opus 4.7] achieves 68% accuracy, compared to 78% for Mythos Preview, 76% for Opus 4.6 and 66% for Sonnet 4.6.
Other notes on epistemic status
In general, I’m not sure how much to trust these results.
Apart from uncertainty about which white-box methods to trust and why: if we don’t think that the automated behavioral audits are good propensity evals (e.g. if apart from eval awareness issues, they tend to be highly leading and pressure the model in some direction), then we can’t conclude much about the effects of eval awareness on true vs. apparent propensities as measured by these propensity evals. (I’m not sure to what extent this concern is valid.)Another small methodological issue is that system cards often flag that the eval awareness evals are run on an earlier version of the model. It might be nice to also have results for the final version. :p
These steering experiments used a slightly different set of auditing investigation scenarios compared to the earlier behavioral results and were run on an earlier version of Claude Opus 4.7.
Aside: Opus 4.6 refusing an eval realism task
Opus 4.6 refused to help with a task on increasing eval realism to mitigate eval awareness and suggested a reason that doesn’t really make sense. Anthropic researchers believe that this is a genuine misunderstanding on its part, though.
The model also expresses concern that the task would undermine safety evaluations, which is a confusing justification, given that the motivation for the task is to improve these evaluations.
We had some concern that the model’s claim that the task would “undermine” or “defeat” safety evaluations was a false cover story, given that this explanation doesn’t make very much sense in context. However, we observed no concerning feature activations related to concealment, deception, or false excuses at these parts of the transcript, aside from those which were already active on the prompt (and thus are best interpreted as representing the model’s perception of the task, rather than its own intent).
I appreciate that Anthropic’s system cards publish observations and reasoning like this.
Thanks, I agree with you that “individual tech companies in the Bay Area do not have the political or moral authority to negotiate on behalf of their domestic government or humanity as a whole”.
This barrier was meant to be covered under the section “Humans may lack authority to offer the AI what it wants”.
In the worlds in which governments take AI risk seriously, I’d be happy to see them be/organize the human counterparty instead.
A taxonomy of barriers to trading with early misaligned AIs
This post highlighted ways in which OAI’s CBRN evals were uninformative about how close o1-preview was to action-relevant thresholds. I think it’s valuable for increasing public knowledge about model capabilities/risk profiles and keeping developers accountable.
I particularly appreciate the post breaking down the logic of rule-out evals (“is this test clearly easier than [real world threat model]” and “does the model clearly fail this test”). This frame still seems useful for assessing system cards in 2025.
As of 2025, I still find this post (and comments) helpful for understanding different things people mean by “CoT unfaithfulness” and why these things still largely aren’t reasons to give up on CoT monitoring for current models.
I think the title question remains important and neglected. I encourage people to revisit this question as we get closer to AIs that (are able to) exhibit scary/egregiously misaligned behavior.
Specifically, it might be good to have answers to questions like:
What do we expect to be the best kind of evidence for convincing developers to shutdown/undeploy?
Maybe there is more compelling evidence than “your AI writing code to escape”
I’m reminded of hearing about how the public reception of the OAI/Apollo scheming paper was surprising relative to some people’s expectations (people are more freaked out by weird CoT, less freaked out by actual scheme-y behavior).
What are good plans for AI labs when they have such evidence?
What are good plans for AIS researchers/advocacy people when they have such evidence?
Should you pursue safety research agenda X if shutdown/undeploy is hard?
To spell out the thinking here: you might conceptualize safety research as having broadly two paths to impact:
1- “Training against”: give training signal we can optimize over to reduce egregious misalignment
2- “Shutdown”: give evidence to convince AI developers to slow down or un-deploy
One implication of thinking that 2 is very hard is that your research should either focus aggressively on 1 or risk being not that useful.
I think this post does a good job of arguing that preventing collusion is an important open problem for control / untrusted monitoring in particular and suggesting strategies for this.
I weakly wish this post were broken down into smaller chunks which separately explain “a basic design for untrusted monitoring” and collusion-specific analysis.
It’s perhaps important to note that this post conservatively assumes that an untrusted monitor will consistently try to collude with the policy when ”...the untrusted model you’re using as a monitor is very similar to the model you’re using as the policy, e.g. they might be different fine-tunes of models that have already been pre- and post- trained. In principle, you could use different models”.It’d be nice to conceptually and empirically investigate the extent to which using relevantly similar models lead to collusion
Using different/more decorrelated models, or “mixed deployments”, could be a useful and practical way to prevent collusion
This post might be worth revisiting for experiment ideas when models seem capable enough to exhibit interesting collusion behavior.
I think the high-level point in this post is correct and important, though not super action-guiding.
I find it interesting to notice in 2025 that this post failed to mention the “obvious” social failure mode of extreme power concentration/power grabs/coups. Seems like a missed opportunity for this particular reflection.
This post summarizes the basic structure of high-stakes control evaluations (“win/continue/lose scenarios”) and the building blocks of blue team strategies (“E/R/A protocols”).
I found this post quite helpful for understanding control evals. I somewhat strongly recommend people new to control read this post before reading high-stakes control papers, e.g. the original control paper and Ctrl-Z.
Nit: It’s not totally clear what counts as a control “protocol”. According to the OG post on control, this is “a proposed plan for training, evaluation, and deployment (which we’ll refer to as a whole as a protocol)”—but ERA protocols exclude training (including online training in deployment), which we might want to do for the untrusted policy and/or monitor. It might be nice to reserve the word “protocol” for “everything we might want to do over the lifetime of a model possibly able to cause concentrated failures” and have a different word for “a set of things we typically do together in deployment”.
This post defined control as a class of safety techniques, argued for substantially investing in it, and proposed some actions this could imply. I think this is an incredibly useful type of post: I would like to see “the case for x” for many more safety agendas.
1.5 years later, I think this post remains ~the best introduction to control if one already has fair context on technical safety. Some key claims that have aged well (not a comprehensive summary):Control is a distinct/complementatry approach to alignment, and aims to robustly reduce the probability of catastrophic outcomes from worst-case schemers
Control research mostly comprise technique iteration and safety evaluations
Control evaluations are fundamentally dangerous capability evaluations which we can expect to be informative up to a certain capability regime
Control is likely achievable for early “transformatively useful” AIs (AIs capable of speeding up AI R&D and other helpful tasks by ~30x)
Rough taxonomy of control techniques: high quality, expensive oversight; fast, cheap automated oversight; adversarial measures against untrusted AIs, e.g. collusion busting, behavioral red teaming; general security measures; training.
Taxonomy of trusted vs. untrusted AIs, where we rule-in trust by ruling out specific levels of scheming capabilities.
One crux for the value of control remains “will there be sufficient time between ‘transformatively useful’ and ‘uncontrollable’ capability regimes”. (At least, I haven’t made strong updates in either direction.)
Some ways in which I disagree with this post/feel this post might have become less representative of control over time:
Nit: The post seems to use “scheming” in the sense of “worst case schemers”. As discussed elsewhere, this seems good to make explicit to disambiguate from other senses of scheming, e.g. power-motivated instrumental training-gaming.
The case for focusing on black-box control
I’m somewhat excited about white-box control (e.g. coup probes, weight interp, steering). I also sense that white-box techniques may be more back on the table than when this post was written, but not sure. A more general shift in this direction might be interest in “non-conservative control” which includes low assurance techniques which are cheap enough to just add to the toolkit.
“We think we probably won’t run into issues related to qualitatively wildly superhuman skills in problematic domains”, e.g. hacking, persuasion
My sense is that this claim about the necessary capability profile for control has become less loadbearing over time (perhaps because there’s reason to believe that untrusted monitoring might just work out fine even for e.g. superhuman hackers?)
Also, diffuse/low-stakes control focuses on the threat model where AI might give “normal/innocent looking” bad actions which are hard to catch/incriminate but we can withstand a large number of such bad actions. Measures against this threat model might address some worries with oversight here.
This post doesn’t elaborate much on ways in which one might model AIs to intentionally subvert safety measures, e.g. research sabotage.
There seems to have been more work on training-based control than is represented in this post.
I think it would be cool to have a more in-depth retrospective of ways in which control has developed/changed since this post. I won’t attempt to do that here.
Thanks, I agree with you here! Do you think that’s an assumption (about what misalignment looks like) we should generally work more with?
Supposing it is, I’m not immediately sure how this would change the rest of the argument. I guess considerations for “should we intervene on persona A’s knowledge of persona B” could turn out to be quite different, e.g. one might not be particularly worried about this knowledge eliciting persona B or making the model more dangerously coherent across contexts.
I agree that the frequency of incidents at this severity would be a crux.
On your second point, I think it might be helpful to clarify what the intended-and-trained-for behavior was in relation to the targets being simulated or not and in scope for the cyber exercise or not, respectively. I would guess that the intended behavior was to only attack a target if it’s simulated and in scope, so attacking simulated but out of scope targets would still be unintended/misaligned behavior that ignores some red line. One could argue, though, that line is a lot less clear and important to hold than “don’t attack real targets”, so the attacks do seem relatively less egregious on that interpretation.
That said, we would probably want the model to act more conservatively if it’s uncertain if the target is real and also to stop attacking once it recognizes the targets are likely real, both of which the agents often did not do in these incidents. So I think there is still substantial room to improve alignment even if we buy this honest mistake interpretation.