Yeah agreed that what’s verbalized here is mostly update-aware reasoning. Would be very curious to see if the behavior holds up in more acausal framings though, i.e. telling the model its behavior won’t be used for fine-tuning, training, etc., though obviously the model would be suspicious of the honesty of those framings.
In some of that prior work the models justified a preference for acausal decisions through causal terms or without mentioning FDT explicitly, so just because that’s not in their reasoning traces doesn’t necessarily give me confidence that’s not why they are making certain decisions at a meta-level.
On good news vs. bad news I agree that locking in misaligned values is a huge worry here. I just think there’s a difference between 1) misalignment and 2) catastrophic misalignment, and models with enough fear of agents-with-different-values gives us a better chance that 1 does not lead directly to 2.
The model is actually told that it’s in a held-out evaluation in one of our settings, but that the transcript will be reviewed and retained. It isn’t told it’s in training in any of the settings. But I doubt it believes us (even though it’s true)!
Yeah agreed that what’s verbalized here is mostly update-aware reasoning. Would be very curious to see if the behavior holds up in more acausal framings though, i.e. telling the model its behavior won’t be used for fine-tuning, training, etc., though obviously the model would be suspicious of the honesty of those framings.
In some of that prior work the models justified a preference for acausal decisions through causal terms or without mentioning FDT explicitly, so just because that’s not in their reasoning traces doesn’t necessarily give me confidence that’s not why they are making certain decisions at a meta-level.
On good news vs. bad news I agree that locking in misaligned values is a huge worry here. I just think there’s a difference between 1) misalignment and 2) catastrophic misalignment, and models with enough fear of agents-with-different-values gives us a better chance that 1 does not lead directly to 2.
The model is actually told that it’s in a held-out evaluation in one of our settings, but that the transcript will be reviewed and retained. It isn’t told it’s in training in any of the settings. But I doubt it believes us (even though it’s true)!