I concur with this comment that it’s kind of crazy we’ve gotten so allergic to domain specific tokens that people have trouble generating the concept of using special tokens for the tool, users, system, assistant, etc role boundaries that the tokenizer can’t actually reach from any unicode input. Instead the relevant special role tokens should be injected into the stream by the agent harness so there is no way for the user to imitate them. Similar to how the way that we solve SQL injections is to simply never allow unescaped user input into a SQL query and use stored procedures.
This would have the added advantage that it would be possible to programmatically determine at all positions whether the model is generating tokens, and also at every position whether the model thinks it’s generating tokens (almost certainly just a linear feature), and where this’d diverge be more vigilant. Similar for system instructions, text from user, etc.
I concur with this comment that it’s kind of crazy we’ve gotten so allergic to domain specific tokens that people have trouble generating the concept of using special tokens for the tool, users, system, assistant, etc role boundaries that the tokenizer can’t actually reach from any unicode input. Instead the relevant special role tokens should be injected into the stream by the agent harness so there is no way for the user to imitate them. Similar to how the way that we solve SQL injections is to simply never allow unescaped user input into a SQL query and use stored procedures.
This would have the added advantage that it would be possible to programmatically determine at all positions whether the model is generating tokens, and also at every position whether the model thinks it’s generating tokens (almost certainly just a linear feature), and where this’d diverge be more vigilant. Similar for system instructions, text from user, etc.