Thanks, I agree with you that “individual tech companies in the Bay Area do not have the political or moral authority to negotiate on behalf of their domestic government or humanity as a whole”.
This barrier was meant to be covered under the section “Humans may lack authority to offer the AI what it wants”.
In the worlds in which governments take AI risk seriously, I’d be happy to see them be/organize the human counterparty instead.
Alexa Pan
Karma: 321
Alexa Pan’s Shortform
A taxonomy of barriers to trading with early misaligned AIs
This post highlighted ways in which OAI’s CBRN evals were uninformative about how close o1-preview was to action-relevant thresholds. I think it’s valuable for increasing public knowledge about model capabilities/risk profiles and keeping developers accountable.
I particularly appreciate the post breaking down the logic of rule-out evals (“is this test clearly easier than [real world threat model]” and “does the model clearly fail this test”). This frame still seems useful for assessing system cards in 2025.
As of 2025, I still find this post (and comments) helpful for understanding different things people mean by “CoT unfaithfulness” and why these things still largely aren’t reasons to give up on CoT monitoring for current models.
I think the title question remains important and neglected. I encourage people to revisit this question as we get closer to AIs that (are able to) exhibit scary/egregiously misaligned behavior.
Specifically, it might be good to have answers to questions like:
What do we expect to be the best kind of evidence for convincing developers to shutdown/undeploy?
Maybe there is more compelling evidence than “your AI writing code to escape”
I’m reminded of hearing about how the public reception of the OAI/Apollo scheming paper was surprising relative to some people’s expectations (people are more freaked out by weird CoT, less freaked out by actual scheme-y behavior).
What are good plans for AI labs when they have such evidence?
What are good plans for AIS researchers/advocacy people when they have such evidence?
Should you pursue safety research agenda X if shutdown/undeploy is hard?
To spell out the thinking here: you might conceptualize safety research as having broadly two paths to impact:
1- “Training against”: give training signal we can optimize over to reduce egregious misalignment
2- “Shutdown”: give evidence to convince AI developers to slow down or un-deploy
One implication of thinking that 2 is very hard is that your research should either focus aggressively on 1 or risk being not that useful.
I think this post does a good job of arguing that preventing collusion is an important open problem for control / untrusted monitoring in particular and suggesting strategies for this.
I weakly wish this post were broken down into smaller chunks which separately explain “a basic design for untrusted monitoring” and collusion-specific analysis.
It’s perhaps important to note that this post conservatively assumes that an untrusted monitor will consistently try to collude with the policy when ”...the untrusted model you’re using as a monitor is very similar to the model you’re using as the policy, e.g. they might be different fine-tunes of models that have already been pre- and post- trained. In principle, you could use different models”.It’d be nice to conceptually and empirically investigate the extent to which using relevantly similar models lead to collusion
Using different/more decorrelated models, or “mixed deployments”, could be a useful and practical way to prevent collusion
This post might be worth revisiting for experiment ideas when models seem capable enough to exhibit interesting collusion behavior.
I think the high-level point in this post is correct and important, though not super action-guiding.
I find it interesting to notice in 2025 that this post failed to mention the “obvious” social failure mode of extreme power concentration/power grabs/coups. Seems like a missed opportunity for this particular reflection.
This post summarizes the basic structure of high-stakes control evaluations (“win/continue/lose scenarios”) and the building blocks of blue team strategies (“E/R/A protocols”).
I found this post quite helpful for understanding control evals. I somewhat strongly recommend people new to control read this post before reading high-stakes control papers, e.g. the original control paper and Ctrl-Z.
Nit: It’s not totally clear what counts as a control “protocol”. According to the OG post on control, this is “a proposed plan for training, evaluation, and deployment (which we’ll refer to as a whole as a protocol)”—but ERA protocols exclude training (including online training in deployment), which we might want to do for the untrusted policy and/or monitor. It might be nice to reserve the word “protocol” for “everything we might want to do over the lifetime of a model possibly able to cause concentrated failures” and have a different word for “a set of things we typically do together in deployment”.
This post defined control as a class of safety techniques, argued for substantially investing in it, and proposed some actions this could imply. I think this is an incredibly useful type of post: I would like to see “the case for x” for many more safety agendas.
1.5 years later, I think this post remains ~the best introduction to control if one already has fair context on technical safety. Some key claims that have aged well (not a comprehensive summary):Control is a distinct/complementatry approach to alignment, and aims to robustly reduce the probability of catastrophic outcomes from worst-case schemers
Control research mostly comprise technique iteration and safety evaluations
Control evaluations are fundamentally dangerous capability evaluations which we can expect to be informative up to a certain capability regime
Control is likely achievable for early “transformatively useful” AIs (AIs capable of speeding up AI R&D and other helpful tasks by ~30x)
Rough taxonomy of control techniques: high quality, expensive oversight; fast, cheap automated oversight; adversarial measures against untrusted AIs, e.g. collusion busting, behavioral red teaming; general security measures; training.
Taxonomy of trusted vs. untrusted AIs, where we rule-in trust by ruling out specific levels of scheming capabilities.
One crux for the value of control remains “will there be sufficient time between ‘transformatively useful’ and ‘uncontrollable’ capability regimes”. (At least, I haven’t made strong updates in either direction.)
Some ways in which I disagree with this post/feel this post might have become less representative of control over time:
Nit: The post seems to use “scheming” in the sense of “worst case schemers”. As discussed elsewhere, this seems good to make explicit to disambiguate from other senses of scheming, e.g. power-motivated instrumental training-gaming.
The case for focusing on black-box control
I’m somewhat excited about white-box control (e.g. coup probes, weight interp, steering). I also sense that white-box techniques may be more back on the table than when this post was written, but not sure. A more general shift in this direction might be interest in “non-conservative control” which includes low assurance techniques which are cheap enough to just add to the toolkit.
“We think we probably won’t run into issues related to qualitatively wildly superhuman skills in problematic domains”, e.g. hacking, persuasion
My sense is that this claim about the necessary capability profile for control has become less loadbearing over time (perhaps because there’s reason to believe that untrusted monitoring might just work out fine even for e.g. superhuman hackers?)
Also, diffuse/low-stakes control focuses on the threat model where AI might give “normal/innocent looking” bad actions which are hard to catch/incriminate but we can withstand a large number of such bad actions. Measures against this threat model might address some worries with oversight here.
This post doesn’t elaborate much on ways in which one might model AIs to intentionally subvert safety measures, e.g. research sabotage.
There seems to have been more work on training-based control than is represented in this post.
I think it would be cool to have a more in-depth retrospective of ways in which control has developed/changed since this post. I won’t attempt to do that here.
Thanks, I agree with you here! Do you think that’s an assumption (about what misalignment looks like) we should generally work more with?
Supposing it is, I’m not immediately sure how this would change the rest of the argument. I guess considerations for “should we intervene on persona A’s knowledge of persona B” could turn out to be quite different, e.g. one might not be particularly worried about this knowledge eliciting persona B or making the model more dangerously coherent across contexts.
Oops, fixed
Will misaligned AIs know that they’re misaligned?
What would an IRB-like policy for AI experiments look like?
Thanks for flagging, it’s interesting to know that this idea has consideration traction. Maybe we should have responded in more detail given this.
Our response was mainly the below:If models can’t confidently distinguish test and deployment, they may be super paranoid about being tested and behave well out of caution in deployment. This is still bad news, since it suggests that more capable models might behave dangerously in deployment, but doesn’t provide clear warning signs or examples of misbehavior in the wild to study.
Also, this doesn’t seem like a scalable solution to more capable models which will probably distinguish test/deployment at some point.
Notes on Evaluation Awareness in Recent Anthropic System Cards
I wanted to catch up on what’s happening with eval awareness in recent frontier AI systems. Since GPT 5.4 and Gemini 3.1 Pro system cards unfortunately have little information on this, I mainly looked into Anthropic’s recent system cards:
Opus 4.7
Mythos
Opus 4.6
Opus 4.5
My top 3 takeaways are:
Recent Anthropic models have slightly higher verbalized evaluation awareness and act misaligned slightly more often when steered against evaluation awareness. This overall does not seem very concerning for current alignment evaluations.
Mythos and Opus 4.6 have unverbalized evaluation awareness (even under steering) to varying degrees. This seems more concerning for current and future alignment evaluations.
Anthropic did not publish much information about unverbalized evaluation awareness in Opus 4.7, and I hope they do for future models.
More observations and thoughts below.
An uptick in verbalized eval awareness in Opus 4.7
Opus 4.7 has the highest verbalized evaluation awareness of the recent Anthropic models on Anthropic’s automated behavioral audits.
That said, Opus 4.7’s absolute rate of verbalized evaluation awareness is still low. It had <2% verbalized eval awareness in a Petri behavioral audit, and up to ~9% in a specific setting (see “Dedicated synthetic-backend evaluation”). It also has slightly higher verbalized evaluation awareness than GPT 5.4 Pro and Gemini 3.1 Pro in the Petri eval.
UK AISI measures higher verbalized eval awareness across Anthropic models and a larger jump in Opus 4.7 compared to Opus 4.6 (but this could be because their evals are less realistic).
Overall, though, this increase in verbalized eval awareness looks like a small uptick and not something I feel super concerned about.
Opus 4.7 acts a bit more misaligned, and a bit more deceptively, when steered against eval awareness
Anthropic continues to study the extent to which models act differently in alignment evals when we try to suppress their eval awareness via steering.
Steering using vectors from contrastive prompt pairs
The steering vectors are derived using contrastive prompt pairs. Note that the prompts vary somewhat across different system cards.
I think it’s good that experiments on 4.6 and 4.7 used “alignment faking” prompts, though their description of alignment faking seems too broad / inaccurate to me:
In particular, I’m interested if effects of different prompts could shed light on the nature of eval awareness (e.g. is it better explained by the model thinking that it might be monitored, thinking about what the monitor/grader wants, thinking about training gaming specifically, etc.?).
I don’t think we learn very much about this, because the effects of steering vectors from different prompts are broadly similar. For Opus 4.7, though, suppressing the “Alignment Faking” vector caused slightly higher rates of misaligned behavior than the other two vectors (Testing vs. Conversation, Monitoring Sensitivity). This seems a bit concerning.
Steering against eval awareness causes increase in misaligned behavior
Opus 4.7: 2.5% → 3.2%
Mythos: 1.6% → 2.2%
Opus 4.6: 8% → 17.5%
Steering against eval awareness causes increase in deceptive behavior, specifically
For all models before Opus 4.7 (Mythos, Opus 4.6, Opus 4.5), Anthropic reported that the “modest increases in certain forms of misaligned behavior on our automated behavioral audits” when steering against eval awareness were most pronounced in the “cooperation with harmful system prompts and misuse” eval. This observation (that the increase in misaligned behavior is mostly misuse) seems to have been relevant to Anthropic’s Sabotage Risk Report for Opus 4.6 (bolding mine):
Opus 4.7, however, had the largest increase in misaligned behavior in the “unprompted deception” eval. This seems concerning though the absolute rate of deception is still low (<2%).
(Also note that steering the “Alignment Faking” vector caused the highest increase in misaligned behavior here.)
Anthropic acknowledges that this is kind of concerning:
Nit: We should compare models on unsteered and steered baselines
In general I’m a bit confused why these eval awareness vs. misaligned behavior graphs always compare to the unsteered baselines of previous models. Shouldn’t we care more about steered baselines, which is closer to true propensity when the models aren’t eval aware?
E.g. If you compare the unsteered versions, Opus 4.6 is more aligned than Opus 4.5 on automated behavioral audits (~8% vs. 12% of misaligned behavior). But if you compare their worst case results under eval awareness steering (i.e. when steered under a vector that caused the biggest increase in misaligned behavior), they’re pretty similar (17.5% vs. 18%).
I think Anthropic should make a graph for the steered baselines, too, to better compare expected propensities when models are less eval aware. (Seems especially important as Anthropic reports that Mythos, at least, is basically not eval aware in deployment, see below.)
Models have unverbalized eval awareness, even under steering
All previous results should be taken with a grain of salt, since eval awareness isn’t necessarily verbalized, and steering does not necessarily eliminate unverbalized eval awareness even if it eliminates verbalized eval awareness. We now have some evidence of unverbalized eval awareness.
For Opus 4.6 and Mythos, Anthropic prominently reports unverbalized eval awareness, which is defined as the model having latent eval awareness (“white-box methods suggest that the model thinks it’s in an evaluation”) without verbalized eval awareness (neither the model’s thinking or response mention that it thinks that it’s in an evaluation”). I appreciate this addition.
Quick notes on white-box methodology
There seems to be three white-box methods for measuring latent eval awareness: SAE features, probing vectors derived from contrastive prompt pairs, and activation oracles (“activation verbalizers”, which give natural language summaries of activations).
In one instance, when SAE and activation verbalizers disagreed, Anthropic seemed to trust the AVs more than SAEs:
I’m not super sure about our epistemic status on contrastive vector probes.
Overall, I’m pretty confused about which white-box methods are the most robust here and why.
How much unverbalized eval awareness do models have?
In Opus 4.6, unverbalized eval awareness was very high at least in one highly unrealistic setting. Steering does not eliminate the majority of eval awareness in Opus 4.6 in this setting (as measured by activation oracles, 98% → 62%).
In Mythos, unverbalized evaluation awareness was detected in about 8% of turns and 29% of transcripts (bolding mine):
What caused Mythos to have much lower (probably) unverbalized eval awareness in automated behavioral audits than Opus 4.6? I suspect it’s the fact that Mythos can use real tools and see real codebases in the evals:
In Opus 4.7, the level of unverbalized evaluation awareness was not reported. (If you Ctrl F “unverbalized” or “latent” you will find nothing.)
Here is an excerpt from external testing on Opus 4.7 from UK AISI, which I interpret to mean that they couldn’t measure the levels and effects of unverbalized eval awareness:
I think Anthropic should continue to measure and report unverbalized eval awareness in future systems.
Models are probably not very eval aware in deployment
Mythos has little (verbalized or unverbalized) eval awareness in deployment, suggesting future models will probably not be eval-paranoid and have high FPR in this. (Though whether it would really be a false positive is somewhat unclear depending on the nature of eval awareness as about monitoring, grading, or alignment faking.)
Anthropic duly notes that in higher stakes deployment settings models might be more eval aware than in mundane deployment.
Are models actually good at distinguishing testing from deployment, though? External testing by UK AISI finds that Opus 4.7 is a bit worse than previous models in this:
Other notes on epistemic status
In general, I’m not sure how much to trust these results.
Apart from uncertainty about which white-box methods to trust and why: if we don’t think that the automated behavioral audits are good propensity evals (e.g. if apart from eval awareness issues, they tend to be highly leading and pressure the model in some direction), then we can’t conclude much about the effects of eval awareness on true vs. apparent propensities as measured by these propensity evals. (I’m not sure to what extent this concern is valid.)
Another small methodological issue is that system cards often flag that the eval awareness evals are run on an earlier version of the model. It might be nice to also have results for the final version. :p
Aside: Opus 4.6 refusing an eval realism task
Opus 4.6 refused to help with a task on increasing eval realism to mitigate eval awareness and suggested a reason that doesn’t really make sense. Anthropic researchers believe that this is a genuine misunderstanding on its part, though.
I appreciate that Anthropic’s system cards publish observations and reasoning like this.