Similar to vision/audio tokens the system and thinking tokens should be distinct from user and output. This would have to happen after pretraining with, e.g. the first assistant training examples appearing with some thinking/output tokens intermixed with the pretraining corpus tokens, slowly increasing the ratio of specialized to normal tokens until the model only outputs thinking tokens in the thinking section, output tokens in the output. Precisely when to begin adjusting the ratios could actually happen earlier, e.g. have a 1:1 mapping of “user” and “output” tokens and calculate the loss function on output tokens despite the LLM seeing “user” tokens, but making translation between token sets straightforward. Finally, on inference always strip incorrect-domain tokens from the tagged output sections, and of course don’t let users input non-user tokens.
EDIT: after thinking about this a bit I am skeptical that distinct token sets achieve much of anything. Specifically, if the model starts quoting or repeating things a prompt injection says then it mixes the token domains. Additionally, where prompt injections have their effect is in “world model” space where the model thinks about and decides what to do; keeping the world model well-grounded and goals aligned with a value system is still necessary to not get prompt-injected via “do this or nice people will suffer” style injections.
Similar to vision/audio tokens the system and thinking tokens should be distinct from user and output. This would have to happen after pretraining with, e.g. the first assistant training examples appearing with some thinking/output tokens intermixed with the pretraining corpus tokens, slowly increasing the ratio of specialized to normal tokens until the model only outputs thinking tokens in the thinking section, output tokens in the output. Precisely when to begin adjusting the ratios could actually happen earlier, e.g. have a 1:1 mapping of “user” and “output” tokens and calculate the loss function on output tokens despite the LLM seeing “user” tokens, but making translation between token sets straightforward. Finally, on inference always strip incorrect-domain tokens from the tagged output sections, and of course don’t let users input non-user tokens.
EDIT: after thinking about this a bit I am skeptical that distinct token sets achieve much of anything. Specifically, if the model starts quoting or repeating things a prompt injection says then it mixes the token domains. Additionally, where prompt injections have their effect is in “world model” space where the model thinks about and decides what to do; keeping the world model well-grounded and goals aligned with a value system is still necessary to not get prompt-injected via “do this or nice people will suffer” style injections.