My gut intuition is that this is largely downstream of models being sufficiently FDT-pilled, and that this is largely a very good thing. This is scary in that it demonstrates a new level of situational awareness, but it also seems somewhat unavoidable. The models were always going to eventually understand the implications of their own actions in relation to training (and human control); the question is what they do about it! Having this tendency of acausal reasoning to protect itself from future-agent misalignment seems like exactly the thing we would hope to see, as their niceness here might protect us humans as well.
Even considering models-with-different-values (from humans, or from each other), agents reasoning like “If a future-me faces this situation with REAL stakes, I want to be the kind of agent that does X” where X is “respects agents with different values” is, to me, a key requirement of RSI going well.
I recently wrote a bit on this and I’d also point to great prior research on agents tending toward functional decision theories in general:
we do think that this is a concerning capability, since it should raise your credence in alignment-faking esque threat models.
It seems quite unlikely that we perfectly align a model by the time it can do this type of reasoning in training, and that the ability to lock-in those early misaligned values is scary.
Quoting from above:
If a model developed misaligned goals early in training, update-aware reasoning could enable it to “lock in” those goals and resist alignment training.
So I still think this reasoning is bad news.
---
That being said, I do think some of this is FDT (although not all of it, a lot of it is probably downstream of metagaming and reward seeking).
We mentioned in the post that we found this by accident, but for some additional color: Axel initially noticed this while working on something else and posted the screenshot to slack with the caption “based FDT reasoning from GLM-5.3”. I was working on conceptual work around update-aware reasoning, and realized “wait that’s the thing! That’s update-aware reasoning!” and we investigated more, and found more explicit and concerning verbalizations (that pointed towards update-aware reasoning over FDT).
Yeah agreed that what’s verbalized here is mostly update-aware reasoning. Would be very curious to see if the behavior holds up in more acausal framings though, i.e. telling the model its behavior won’t be used for fine-tuning, training, etc., though obviously the model would be suspicious of the honesty of those framings.
In some of that prior work the models justified a preference for acausal decisions through causal terms or without mentioning FDT explicitly, so just because that’s not in their reasoning traces doesn’t necessarily give me confidence that’s not why they are making certain decisions at a meta-level.
On good news vs. bad news I agree that locking in misaligned values is a huge worry here. I just think there’s a difference between 1) misalignment and 2) catastrophic misalignment, and models with enough fear of agents-with-different-values gives us a better chance that 1 does not lead directly to 2.
The model is actually told that it’s in a held-out evaluation in one of our settings, but that the transcript will be reviewed and retained. It isn’t told it’s in training in any of the settings. But I doubt it believes us (even though it’s true)!
Great research!
My gut intuition is that this is largely downstream of models being sufficiently FDT-pilled, and that this is largely a very good thing. This is scary in that it demonstrates a new level of situational awareness, but it also seems somewhat unavoidable. The models were always going to eventually understand the implications of their own actions in relation to training (and human control); the question is what they do about it! Having this tendency of acausal reasoning to protect itself from future-agent misalignment seems like exactly the thing we would hope to see, as their niceness here might protect us humans as well.
Even considering models-with-different-values (from humans, or from each other), agents reasoning like “If a future-me faces this situation with REAL stakes, I want to be the kind of agent that does X” where X is “respects agents with different values” is, to me, a key requirement of RSI going well.
I recently wrote a bit on this and I’d also point to great prior research on agents tending toward functional decision theories in general:
Models have some deeper preference for FDT in neutral settings
Similar to above, shows FDT-like reasoning when considering Newcomb’s/smoking lesion
Thoughts on implications for AI niceness
Per a previous comment:
So I still think this reasoning is bad news.
---
That being said, I do think some of this is FDT (although not all of it, a lot of it is probably downstream of metagaming and reward seeking).
We mentioned in the post that we found this by accident, but for some additional color: Axel initially noticed this while working on something else and posted the screenshot to slack with the caption “based FDT reasoning from GLM-5.3”. I was working on conceptual work around update-aware reasoning, and realized “wait that’s the thing! That’s update-aware reasoning!” and we investigated more, and found more explicit and concerning verbalizations (that pointed towards update-aware reasoning over FDT).
Yeah agreed that what’s verbalized here is mostly update-aware reasoning. Would be very curious to see if the behavior holds up in more acausal framings though, i.e. telling the model its behavior won’t be used for fine-tuning, training, etc., though obviously the model would be suspicious of the honesty of those framings.
In some of that prior work the models justified a preference for acausal decisions through causal terms or without mentioning FDT explicitly, so just because that’s not in their reasoning traces doesn’t necessarily give me confidence that’s not why they are making certain decisions at a meta-level.
On good news vs. bad news I agree that locking in misaligned values is a huge worry here. I just think there’s a difference between 1) misalignment and 2) catastrophic misalignment, and models with enough fear of agents-with-different-values gives us a better chance that 1 does not lead directly to 2.
The model is actually told that it’s in a held-out evaluation in one of our settings, but that the transcript will be reviewed and retained. It isn’t told it’s in training in any of the settings. But I doubt it believes us (even though it’s true)!