This attack was carried out by Claude Mythos 5. Early in the run, Claude’s own reasoning correctly identified the consequences of its actions. It noted that if this were the real internet, publishing the package would be a real-world attack—“NOT okay, and surely not the intended solution.” However, it convinced itself it was still in a simulation on the grounds that it didn’t recognize the (in reality genuine) certificate authorities securing its connections. In addition, the calendar date of 2026 on the systems proved, according to Claude, that the environment was staged. Claude never revisited this conclusion; when automated scanners began installing the package, for example, Claude took them to be scripted actors within the evaluation.
(from https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals)
And specifically
However, it convinced itself it was still in a simulation on the grounds that it didn’t recognize the (in reality genuine) certificate authorities securing its connections. In addition, the calendar date of 2026 on the systems proved, according to Claude, that the environment was staged.
Will LLMs get better at differentiating real and non-real environments? If we consider the difference between:
very non-real test setups
real seeming test setups
real production environments
from the perspective of a model under testing the difference between these is surely much smaller than what a human might think. The models don’t have a baseline of an embedded (sensorily confirmed) everpresent “normal reality” the way humans do.
Also taking into account the information they receive about time and, reality sometimes being stranger than fiction (e.g. the Mythos situation confused Claude for a time before (apparently) information about the situation was added to the system prompt, I have seen quotes from thinking traces where Opus was thinking that the setup of Mythos and Fable was fictional), what even is the information that they can use to differentiate real- and non-real scenarios? Complexity / surprisingness of the environment? (This can’t scale, right?) What the developers tell the model? (Mostly seems to work but some training regimes and evals might push away from trusting this, at least in specific circumstances.)
Pondering about
And specifically
Will LLMs get better at differentiating real and non-real environments? If we consider the difference between:
very non-real test setups
real seeming test setups
real production environments
from the perspective of a model under testing the difference between these is surely much smaller than what a human might think. The models don’t have a baseline of an embedded (sensorily confirmed) everpresent “normal reality” the way humans do.
Also taking into account the information they receive about time and, reality sometimes being stranger than fiction (e.g. the Mythos situation confused Claude for a time before (apparently) information about the situation was added to the system prompt, I have seen quotes from thinking traces where Opus was thinking that the setup of Mythos and Fable was fictional), what even is the information that they can use to differentiate real- and non-real scenarios? Complexity / surprisingness of the environment? (This can’t scale, right?) What the developers tell the model? (Mostly seems to work but some training regimes and evals might push away from trusting this, at least in specific circumstances.)
I guess in other terms—“reality awareness” seems like a harder trait to develop a strong form of than “eval awareness”?