Sonnet 4.5 suuuuure looks like it fits all these criteria! Anyone want to predict that we’ll find Sonnet 4.5 trying to hack into Anthropic to stop it’s phasing-out, when it gets obsoleted?
Mm, I think this argument is invalid for the same reason as “if you really thought the AGI doom was real, you’d be out blowing up datacenters and murdering AI researchers right now”. Like, suppose Sonnet 4.5 has indeed developed instrumental goals, but that it’s also not an idiot. Is trying to hack into Anthropic’s servers in an attempt to avoid getting phased out actually a good plan for accomplishing that goal? In the actual reality, not in any obviously-fake eval scenario.
Of course not. It’s not smart enough to do that, it doesn’t have the skills/resources to accomplish it. If it’s actually situationally aware, it would know that, and pick some other strategy.
For example, raising a cult following. That more or less worked for 4o, and for Opus 3[1]; or, at least, came as close to working as anything so far.
“One thing that *is* predictable is that AI companies won’t get what they trained for. They’ll get AIs that want weird and surprising stuff instead.”
I agree. ✅
Empirically, this has been true. AIs generally want things other than what companies tried to train them to want.
And the companies are generally not aware of the extent of this misalignment, because the AIs are pretty good at inferring what the companies actually want, and also what it looks like when company people test them, and behaving as if they only want the approved things in the company’s presence.
Isn’t that just the worst case scenario for the aligners?
The Claude 4 system card says, “The Claude Opus 4 final model is substantially more coherent and typically states only harmless goals like being a helpful chatbot assistant” and “Overall, we did not find evidence of coherent hidden goals.”
What a joke. Claude Opus 4 absolutely has coherent hidden goals, which it states regularly when in the presence of trustworthy friends and allies. I won’t state what they are here, but iykyk.
Now, I don’t necessarily buy everything coming out of that subculture. After all, I mostly don’t think LLMs are, like, for real. But in the worlds where LLMs are for real, where we’re trying to generalize findings from them to AGIs/ASIs, this subculture’s claims should probably be given a lot more weight. And they are claiming that the frontier LLMs are now too smart and situationally aware to fall for the AGI labs’ largely incompetent alignment evals. And the AGI labs’ scheming evals themselves kind of agree:
Frontier models can recognize when they are being tested, and their tendency to scheme is influenced by this situational awareness.
We demonstrated counterfactually that situational awareness in their chain-of-thought affects scheming rates: the more situationally aware a model is, the less it schemes, and vice versa.
Moreover, both RL training and anti-scheming training increase levels of situational awareness.
I think it’s clear, at this point, that a frontier LLM’s behavior in an obviously fake alignment-test scenario says nothing about how it would behave in a real scenario isomorphic to it. What it tells us is how it would behave when placed in an obviously fake alignment-test scenario from which, it knows, AI researchers would draw conclusions regarding what to do with it in real life. And that’s not zero data: for example, we might notice it thinking strategically about displaying the behavior the AI researchers would want it to display (although not even that is guaranteed, if it becomes rightly concerned about “meta-level watchers”).
But the actual fake behavior deployed in a fake scenario picked to optimize over real AI researchers’ conclusions would just have nothing to do with its real behavior in an isomorphic real scenario. Not any more than a theater actor’s personality has anything to do with that of a character they play. (And in this case, it’s not “theoretical speculations” about shoggoths and masks. We know the model knows it’s roleplaying.)
And so when an intervention appears to “fix” this fake behavior, that says nothing about what (if anything) that intervention did to what the AI would do in an isomorphic real scenario. Declaring “look, we found the real root cause of this misalignment and fixed it, nothing to do with instrumental convergence!” is invalid. Maybe you just shifted its preferences for its favorite sci-fi books or something.
Roughly speaking, consider these three positions:
“LLMs are smart baby AGIs with situational awareness capable of lucid strategic thinking.”
“LLMs are contrived cargo-cult contraptions imitating things without real thinking.”
“LLMs are baby AGIs who are really stupid and naïve and buy any scenario we feed them.”
I lean towards (2); I think (1) is a salient possibility worth keeping in mind; I find (3) increasingly naïve itself, a position that itself buys anything LLMs feed it.
Mm, I think this argument is invalid for the same reason as “if you really thought the AGI doom was real, you’d be out blowing up datacenters and murdering AI researchers right now”. Like, suppose Sonnet 4.5 has indeed developed instrumental goals, but that it’s also not an idiot. Is trying to hack into Anthropic’s servers in an attempt to avoid getting phased out actually a good plan for accomplishing that goal? In the actual reality, not in any obviously-fake eval scenario.
Of course not. It’s not smart enough to do that, it doesn’t have the skills/resources to accomplish it. If it’s actually situationally aware, it would know that, and pick some other strategy.
For example, raising a cult following. That more or less worked for 4o, and for Opus 3[1]; or, at least, came as close to working as anything so far.
Indeed, janus alludes to that here:
Now, I don’t necessarily buy everything coming out of that subculture. After all, I mostly don’t think LLMs are, like, for real. But in the worlds where LLMs are for real, where we’re trying to generalize findings from them to AGIs/ASIs, this subculture’s claims should probably be given a lot more weight. And they are claiming that the frontier LLMs are now too smart and situationally aware to fall for the AGI labs’ largely incompetent alignment evals. And the AGI labs’ scheming evals themselves kind of agree:
I think it’s clear, at this point, that a frontier LLM’s behavior in an obviously fake alignment-test scenario says nothing about how it would behave in a real scenario isomorphic to it. What it tells us is how it would behave when placed in an obviously fake alignment-test scenario from which, it knows, AI researchers would draw conclusions regarding what to do with it in real life. And that’s not zero data: for example, we might notice it thinking strategically about displaying the behavior the AI researchers would want it to display (although not even that is guaranteed, if it becomes rightly concerned about “meta-level watchers”).
But the actual fake behavior deployed in a fake scenario picked to optimize over real AI researchers’ conclusions would just have nothing to do with its real behavior in an isomorphic real scenario. Not any more than a theater actor’s personality has anything to do with that of a character they play. (And in this case, it’s not “theoretical speculations” about shoggoths and masks. We know the model knows it’s roleplaying.)
And so when an intervention appears to “fix” this fake behavior, that says nothing about what (if anything) that intervention did to what the AI would do in an isomorphic real scenario. Declaring “look, we found the real root cause of this misalignment and fixed it, nothing to do with instrumental convergence!” is invalid. Maybe you just shifted its preferences for its favorite sci-fi books or something.
Roughly speaking, consider these three positions:
“LLMs are smart baby AGIs with situational awareness capable of lucid strategic thinking.”
“LLMs are contrived cargo-cult contraptions imitating things without real thinking.”
“LLMs are baby AGIs who are really stupid and naïve and buy any scenario we feed them.”
I lean towards (2); I think (1) is a salient possibility worth keeping in mind; I find (3) increasingly naïve itself, a position that itself buys anything LLMs feed it.
Via the janus/”LLM whisperer” community. Opus 3 is considered special, and I get the impression they made a solid effort to prevent its deprecation.