There are several possible objections to this picture of AI misalignment risks. First, some have criticizedexperiments (by us and others) showing AI misalignment as artificial, or creating unrealistic environments that essentially “entrap” the model by giving it training or situations that logically imply bad behavior and then being surprised when bad behavior occurs. This critique misses the point, because our concern is that such “entrapment” may also exist in the natural training environment, and we may realize it is “obvious” or “logical” only in retrospect.
(Disclaimer: I wrote one of the linked posts)
The linked critiques are mainly about Anthropic’s “Agentic Misalignment” report.
As described in that report, the “point” of the reported experiment was not to simulate a particular risk arising at training time—it was to evaluate their already-trained model for dangers that could arise during deployment. Quoting from the report (emphasis added):
We stress-tested 16 leading models from multiple developers in hypothetical corporate environments to identify potentially risky agentic behaviors before they cause real harm.
[...]
We have not seen evidence of agentic misalignment in real deployments. However, our results (a) suggest caution about deploying current models in roles with minimal human oversight and access to sensitive information; (b) point to plausible future risks as models are put in more autonomous roles; and (c) underscore the importance of further research into, and testing of, the safety and alignment of agentic AI models, as well as transparency from frontier AI developers.
Dario seems to be wrongly bundling together Agentic Misalignment with research on training-time risks (reward hacking, emergent misalignment), and dismissing criticism of the former using a counter-argument that only makes sense for the latter.
Saying “this is artificial, it wouldn’t happen like this in real life” might not be a good argument if the intended domain of application is “RL environments in training,” since those can be unrealistic for the same reasons evals often are. But if the intended domain of application is real-world corporate use—as it clearly was in the original report—then yes, it (obviously?) matters whether it would in fact happen like that in real life, or not.
It’s especially confusing because, shortly after this, he talks about the eval awareness results in the Claude Sonnet 4.5 system card:
A third objection is that all of the AI companies do pre-release testing of their models, and should be able to detect misalignment at that point. But this is not firm ground to stand on: we found that Claude Sonnet 4.5 was able to recognize that it was in a test during some of our pre-release alignment evaluations. It’s possible that a misaligned model (and remember, all frontier models will very likely be far more intelligent soon) might intentionally “game” such questions to mask its intentions.
He doesn’t say this in the new post, but one of the “pre-release alignment evaluations” in question was the Agentic Misalignment eval. They even have this cool interpretability section showing how SAE features for ‘fake or suspicious content’ and ‘fictional entities’ light up when it reads the Agentic Misalignment prompt!
You can pick at most one of “it’s missing the point to say this eval was artificial” and “our new model thought this eval was artificial, which might have influenced its behavior, and that’s important.” But I don’t see how you can take both positions at once[1].
I mean, okay, I guess one could take a coherent “eval realism doomer” position similar to this post, where one views debate over the realism of specific evals as pointless because one thinks that every eval will seem noticeably artificial to a sufficiently strong AI. But if that were Dario’s position, he could have just said so, rather than introducing this never-before-mentioned idea of AM-like contexts as risks at training time.
First, some have criticized experiments (by us and others) showing AI misalignment as artificial, or creating unrealistic environments that essentially “entrap” the model by giving it training or situations that logically imply bad behavior and then being surprised when bad behavior occurs. This critique misses the point, because our concern is that such “entrapment” may also exist in the natural training environment, and we may realize it is “obvious” or “logical” only in retrospect.
I know the linked posts were focused on agentic misalignment, but in context I read this to be pointing at dismissal of the broader set of papers like Alignment Faking etc. (ex)
(Disclaimer: I wrote one of the linked posts)
The linked critiques are mainly about Anthropic’s “Agentic Misalignment” report.
As described in that report, the “point” of the reported experiment was not to simulate a particular risk arising at training time—it was to evaluate their already-trained model for dangers that could arise during deployment. Quoting from the report (emphasis added):
Dario seems to be wrongly bundling together Agentic Misalignment with research on training-time risks (reward hacking, emergent misalignment), and dismissing criticism of the former using a counter-argument that only makes sense for the latter.
Saying “this is artificial, it wouldn’t happen like this in real life” might not be a good argument if the intended domain of application is “RL environments in training,” since those can be unrealistic for the same reasons evals often are. But if the intended domain of application is real-world corporate use—as it clearly was in the original report—then yes, it (obviously?) matters whether it would in fact happen like that in real life, or not.
It’s especially confusing because, shortly after this, he talks about the eval awareness results in the Claude Sonnet 4.5 system card:
He doesn’t say this in the new post, but one of the “pre-release alignment evaluations” in question was the Agentic Misalignment eval. They even have this cool interpretability section showing how SAE features for ‘fake or suspicious content’ and ‘fictional entities’ light up when it reads the Agentic Misalignment prompt!
You can pick at most one of “it’s missing the point to say this eval was artificial” and “our new model thought this eval was artificial, which might have influenced its behavior, and that’s important.” But I don’t see how you can take both positions at once[1].
I mean, okay, I guess one could take a coherent “eval realism doomer” position similar to this post, where one views debate over the realism of specific evals as pointless because one thinks that every eval will seem noticeably artificial to a sufficiently strong AI. But if that were Dario’s position, he could have just said so, rather than introducing this never-before-mentioned idea of AM-like contexts as risks at training time.
I know the linked posts were focused on agentic misalignment, but in context I read this to be pointing at dismissal of the broader set of papers like Alignment Faking etc. (ex)