I don’t think anything in their training incentivizes self-modeling of this kind.
Safety training may incentivize them to self-model about e.g. “is this the kind of thing that I would say” or “does what I said match what I intended to say”:
Claude models are trained to participate in a dialogue between a human (the user) and an Assistant character, whose outputs the model is responsible for producing. However, users can also prefill the Assistant’s responses, effectively putting words in its mouth. Prefills are a common jailbreaking tactic, and can for instance be used to guide the Assistant to adopt different characteristics, or comply with requests that it would otherwise refuse. However, models are trained to be resilient to such tactics; as a result, the Assistant is reasonably skilled at detecting outputs that are “out of character” for it, and pivoting away from them. [...]
How do models distinguish between their own responses and words placed in their mouth? Doing so must involve estimating the likelihood that the model would have produced a given output token, given the prior context. Broadly, this could be achieved in two ways: (1) the model might ignore its previous intent and recompute what it would have said from raw inputs, or (2) it might directly introspect on its previously computed “intentions”–a representation of its predicted output. There is a spectrum between these extremes (the model can attend to any representation between the raw inputs and later-layer representations of “intent”).
Safety training may incentivize them to self-model about e.g. “is this the kind of thing that I would say” or “does what I said match what I intended to say”: