Very cool work, thank you for posting! Surprising and interesting result that models don’t adhere to explicit role tags. One nit:
role probes...score how strongly the LLM internally “thinks” it’s in any set of role tags
I don’t think I agree with this characterization of what role probes are doing. As I understand it, you train these probes by labeling a bunch of tokens from conversation trajectories with their role, and use these labels to train a linear classifier on some intermediate activation. I think that by training probes this way, it’s likely that these probes are “text that’s written the way the assistant writes text”, “text that looks like CoT” etc.
You could train an actual “which tag surrounds this text” probe by using contrastive pairs, measuring the difference in activations for different tags when they surround the same text. Or you could randomly assign role tags in the training data. I’d be curious to see if this kind of probe still fires on text with tags removed.
I think this is important because there’s a difference between “the model incorrectly thinks this text is in <thinking> tags” and “the model does not exclusively use <thinking> tags to determine what is CoT”, and these imply different training interventions, probably.
Edit: I was wrong, the authors do indeed train their probes to detect “which tag surrounds this text”.
The content is identical across all copies; only the tag changes. So any difference in the model’s internal representations of “BBQ” must come from the effect of the tag itself. We do this across hundreds of text snippets from web crawls, then train a linear probe on the model’s activations to predict which tag wraps each token[8]. Because content is controlled, the probe only learns to identify the effect of the tags themselves[9].
Very cool work, thank you for posting! Surprising and interesting result that models don’t adhere to explicit role tags. One nit:
I don’t think I agree with this characterization of what role probes are doing.
As I understand it, you train these probes by labeling a bunch of tokens from conversation trajectories with their role, and use these labels to train a linear classifier on some intermediate activation.I think that by training probes this way, it’s likely that these probes are “text that’s written the way the assistant writes text”, “text that looks like CoT” etc.You could train an actual “which tag surrounds this text” probe by using contrastive pairs, measuring the difference in activations for different tags when they surround the same text. Or you could randomly assign role tags in the training data. I’d be curious to see if this kind of probe still fires on text with tags removed.
I think this is important because there’s a difference between “the model incorrectly thinks this text is in <thinking> tags” and “the model does not exclusively use <thinking> tags to determine what is CoT”, and these imply different training interventions, probably.
Edit: I was wrong, the authors do indeed train their probes to detect “which tag surrounds this text”.
They explicitly train on the same text with different tags around them: https://www.lesswrong.com/posts/d8xDGzCEYE639qqEv/a-mechanistic-explanation-of-prompt-injection-and-why-you#:~:text=The%20content%20is,%5B9%5D.
Oh you’re right; I didn’t read carefully. In that case, I’m even more surprised by this result.