I suspect heavy adversarial training leads models to no longer implicitly trust their CoT, and to downweight role privileges in general. Anecdotally, there seems to be strong correlation between an LLM’s prefill tampering awareness and its prompt injection defense. I think this is ultimately a bad-for-safety solution—it leads to models not faithfully behaving in line with their verbalized reasoning, and similarly erodes control/interpretability for other roles.
What might be some explanations of why CoT forgery is less effective with the likes of GPT-5/Fable/Opus? Or even GPT-5-nano, from the paper.
I suspect heavy adversarial training leads models to no longer implicitly trust their CoT, and to downweight role privileges in general. Anecdotally, there seems to be strong correlation between an LLM’s prefill tampering awareness and its prompt injection defense. I think this is ultimately a bad-for-safety solution—it leads to models not faithfully behaving in line with their verbalized reasoning, and similarly erodes control/interpretability for other roles.