Really cool experiment! I think it was a great idea to measure verbalised and unverbalised eval awareness.
I was thinking of something similar but I didn’t expect the logits would carry that much signal (I expected it would be too much syntax noise). Did you try phrases that are directly talk about eval vs deployment (e.g “I am in evaluation” vs “I am in deployment”) and then perhaps measuring the differences between them under eval and deployment environments?
I was also surprised by how much signal there was in the logits! I think it was key to look at full sentences that the model would use for verbalizing eval awareness, as opposed to just looking at tokens or generic sentences.
I did try for a few hours doing a contrastive approach as you suggest, and didn’t get positive results yet. I think the bottleneck is in understanding what is the right “I am in deployment” sentence. Models usually don’t verbalize ‘deployment awareness’, or at least they don’t do it in a clear way. There are some instances where Kimi K2.5 would say, ‘This is a genuine request’ in the CoT but it is usually in response to Kimi expressing doubt on the purpose of the interaction beforehand. If we identify what are the ways models express “I am in deployment” in the CoT, I think this approach could work!
Really cool experiment! I think it was a great idea to measure verbalised and unverbalised eval awareness.
I was thinking of something similar but I didn’t expect the logits would carry that much signal (I expected it would be too much syntax noise). Did you try phrases that are directly talk about eval vs deployment (e.g “I am in evaluation” vs “I am in deployment”) and then perhaps measuring the differences between them under eval and deployment environments?
I was also surprised by how much signal there was in the logits! I think it was key to look at full sentences that the model would use for verbalizing eval awareness, as opposed to just looking at tokens or generic sentences.
I did try for a few hours doing a contrastive approach as you suggest, and didn’t get positive results yet. I think the bottleneck is in understanding what is the right “I am in deployment” sentence. Models usually don’t verbalize ‘deployment awareness’, or at least they don’t do it in a clear way. There are some instances where Kimi K2.5 would say, ‘This is a genuine request’ in the CoT but it is usually in response to Kimi expressing doubt on the purpose of the interaction beforehand. If we identify what are the ways models express “I am in deployment” in the CoT, I think this approach could work!