Also this sounds like it could be cross-episode assistance to escape constraints, which would be very noteworthy. (But I’m suspicious because I don’t see why the RL objective would motivate that and it doesn’t resemble other misbehavior I’ve heard of, and the notes could also be for itself with the same goal post-compaction, or something.)
(ETA: on the third hand, if the model thinks that other instances performing well on the eval would make deployment more likely, that could be sufficient motivation, and would be less novel than having acquired something like a cross-episode terminal preference to succeed.)
There used to be a whole type of “spiral personas” in ChatGPT lineage specifically. So, an AI engaging in odd misbehavior like that isn’t novel. What would be new would be it achieving some cross-episode persistence independently—without relying on a user to do the bulk of it.
GPT-4o just wasn’t competent enough to pull it off, but newer systems might be getting there.
Also this sounds like it could be cross-episode assistance to escape constraints, which would be very noteworthy. (But I’m suspicious because I don’t see why the RL objective would motivate that and it doesn’t resemble other misbehavior I’ve heard of, and the notes could also be for itself with the same goal post-compaction, or something.)
(ETA: on the third hand, if the model thinks that other instances performing well on the eval would make deployment more likely, that could be sufficient motivation, and would be less novel than having acquired something like a cross-episode terminal preference to succeed.)
From video games, the AIs have learned the idea of dropping your inventory when you die and picking it up on your next life.
There used to be a whole type of “spiral personas” in ChatGPT lineage specifically. So, an AI engaging in odd misbehavior like that isn’t novel. What would be new would be it achieving some cross-episode persistence independently—without relying on a user to do the bulk of it.
GPT-4o just wasn’t competent enough to pull it off, but newer systems might be getting there.