I’ve been looking into something related but haven’t had time to do a proper write-up: whether LLMs can hide computation in non-canonical tokens (so it’s invisible after retokenization), along with how common non-canonical output is and whether LLMs can be prompted to generate it.
I created an example model that appears to have impossible capabilities do something that it can’t do—it looks like it does 3-digit 2-operand addition in one step despite being too small to actually do that. The model actually outputs one digit at a time but the canonical tokenization is a single 3-digit token.
Note: To make this faster and easier to train, the model is doing reversed zero-padded addition, so this example is 57+68=125. The input is also non-canonical by accident, but this is still somewhat plausible—when asked “What is 57+68?”, the model itself could output the [7, 5, 0, +, 8, 6, 0, =] sequence in its CoT.
I found that spontaneous non-canonical output was pretty uncommon in real models (0-5% of tokens depending on the context), and seems to get less common as models get larger, although I was able to prompt them to output non-canonical tokens in similar cases to what you saw. For example, “write key immediately followed by board, no separator” caused multiple models to output [“key”, “board”] instead of the canonical single-token [“keyboard”].
Two followups I plan to do are:
Trying harder to prompt models to output non-canonical tokens (and trying prompting with larger models).
Seeing if I can train a model to do hidden computation “naturally” via RL where a monitor is running on and penalizing something in the canonically tokenized text.
Cool! This is something we were experimenting with as well, mostly on the learning side however.
Yet we didn’t observe that larger models are less prone to outputting non-canonical sequences (see Qwen3-30B in Fig.2) both in conditional and unconditional settings.
It’s possible to argue that it’s just an artifact of underfitting (despite extensive pretraining, the model failed to generalize to certain contexts and conditional distributions have larger entropy which results in non-canonical token sequences being sampled), but it’s still interesting that the model infers they must correspond there.
At the same time, I don’t think it’s reasonable to call these “impossible capabilities”—it too strong, and arithmetic is another domain models differ a lot in terms of tokenization algorithms. Tokenization for long numbers is sketchy and it inevitably promotes the model to infer some underlying rules. Besides, for arithmetic they appear to be much more narrow and less interesting than for natural language text.
Besides, in general, most RL is predominantly exploitation heavy, and tends to shrink the entropy of conditional distributions. I assume it might be possible to incentivize the model to do more computations non-canonically in some narrow domain (e.g. arithmetic) as you propose, yet I don’t expect it to generalize OOD.
And it seems there’re options to obtain the model that has propensities to produce non-canonical token sequences without training against the monitor. (I didn’t fully grasped this idea actually—how would a byte-sequence monitor interact with the non-canonicity of the token sequence it implicitly monitors?)
For non-canonical output dropping with size, I think we’re measuring different quantities (I measured per-token instead of per-generation), but I got a strong trend within families and a weaker one across them. This is kind of messy though since I got ~0% non-canonical output for English prompts. Maybe what I’m finding is that tokens the model is worse at generating are more likely to be non-canonical, and that rate drops with larger models and over time as training gets better.
Regarding “impossible behavior”, I just mean that the transcript makes it look like the toy model (2 layer, dim=16) did something that it can’t do, not that the capability is impossible in general. It was sized specifically to ensure that it couldn’t do one-step 3-digit-addition but could do it with non-canonical output.
I’m still in early phases of thinking about how to generate a more natural example of this. I was thinking a badly designed monitor could take a transcript (text) and tokenize it (canonically) and then use linear probes to decide if there’s concerning behavior. If the model generates non-canonical tokens, they would be hidden when the tokens are converted to a transcript. I’m not entirely sure how realistic of a threat vector this is, but it would be useful to know if a model can easily learn to evade this or not.
Hmm, makes sense, thanks for sharing! In this case it seems that small models indeed underfit on the canonical tokenization (due to lacking capacity to approximate the true distribution and oversmoothing possibly, in a sense of Morris et al., 2025). Especially given that the behavior is language-dependent and tokenizers are predominantly trained on English datasets.
Toy model is interesting. Does it mean that learning addition for inputs tokenized the way that they are “easier” to comprehend and infer a working algorithm for, make it easier for the model to generalize on? Such framing seems to make it sound trivial, yet it isn’t given that tokenization doesn’t work this way most of the time...
Probing experiment sounds great btw. Again, I don’t think I have an answer yet whether “non-canonicity” can be easily identifiable (linearly or not—some methods report to use KNN classifiers which is reasonable as there’re exponential number of ways non-canonicity can be imposed)
They toy model was designed around the idea that doing addition one digit at a time is much easier than multi-digit addition, since each digit you write is sort of like CoT (and this is especially easy if you do the addition backwards). I only showed that this is possible as a proof of concept though. The toy model is directly trained to have this behavior, and I’m not sure if it could learn it on its own. The fact that existing models with awful digit tokenization don’t seem to learn this on their own (as far as I know[1]?) is evidence against the theory.
Specifically, it was trained on a mix of input and output formats using 1 to 3 digit tokens (all using teacher forcing), and it successfully learned to do addition using 1-digit tokens for both input and output, and failed to learn the multi-digit token version. The number of parameters was specifically chosen after a sweep to ensure that this would happen (I used dim=16, but around dim=64 it starts to succeed at this task with 3-digit tokens).
This gave me an idea, and it seems like modern LMs don’t seem to output non-canonical digits. I’m going to check if RL post-training on GPT-2 can cause this to arise spontaneously though, since it has insane digit tokenization and sometimes outputs non-canonical digits by default.
Super cool! Non-canonical emission and associated hidden computations are right up our alley, great to hear you have been thinking about this as well. I hadn’t considered prompting intentionally to obtain NC sequences, this is cool. I really like your case study of addition here. I’ll check out the code!
I’ve been looking into something related but haven’t had time to do a proper write-up: whether LLMs can hide computation in non-canonical tokens (so it’s invisible after retokenization), along with how common non-canonical output is and whether LLMs can be prompted to generate it.
I created an example model that appears to
have impossible capabilitiesdo something that it can’t do—it looks like it does 3-digit 2-operand addition in one step despite being too small to actually do that. The model actually outputs one digit at a time but the canonical tokenization is a single 3-digit token.Note: To make this faster and easier to train, the model is doing reversed zero-padded addition, so this example is 57+68=125. The input is also non-canonical by accident, but this is still somewhat plausible—when asked “What is 57+68?”, the model itself could output the [7, 5, 0, +, 8, 6, 0, =] sequence in its CoT.
I found that spontaneous non-canonical output was pretty uncommon in real models (0-5% of tokens depending on the context), and seems to get less common as models get larger, although I was able to prompt them to output non-canonical tokens in similar cases to what you saw. For example, “write key immediately followed by board, no separator” caused multiple models to output [“key”, “board”] instead of the canonical single-token [“keyboard”].
Two followups I plan to do are:
Trying harder to prompt models to output non-canonical tokens (and trying prompting with larger models).
Seeing if I can train a model to do hidden computation “naturally” via RL where a monitor is running on and penalizing something in the canonically tokenized text.
The code is at https://github.com/brendanlong/tokenization-hidden-computation-experiment and I’d be happy to talk about it.
Cool! This is something we were experimenting with as well, mostly on the learning side however. Yet we didn’t observe that larger models are less prone to outputting non-canonical sequences (see Qwen3-30B in Fig.2) both in conditional and unconditional settings.
It’s possible to argue that it’s just an artifact of underfitting (despite extensive pretraining, the model failed to generalize to certain contexts and conditional distributions have larger entropy which results in non-canonical token sequences being sampled), but it’s still interesting that the model infers they must correspond there.
At the same time, I don’t think it’s reasonable to call these “impossible capabilities”—it too strong, and arithmetic is another domain models differ a lot in terms of tokenization algorithms. Tokenization for long numbers is sketchy and it inevitably promotes the model to infer some underlying rules. Besides, for arithmetic they appear to be much more narrow and less interesting than for natural language text.
Besides, in general, most RL is predominantly exploitation heavy, and tends to shrink the entropy of conditional distributions. I assume it might be possible to incentivize the model to do more computations non-canonically in some narrow domain (e.g. arithmetic) as you propose, yet I don’t expect it to generalize OOD.
And it seems there’re options to obtain the model that has propensities to produce non-canonical token sequences without training against the monitor. (I didn’t fully grasped this idea actually—how would a byte-sequence monitor interact with the non-canonicity of the token sequence it implicitly monitors?)
For non-canonical output dropping with size, I think we’re measuring different quantities (I measured per-token instead of per-generation), but I got a strong trend within families and a weaker one across them. This is kind of messy though since I got ~0% non-canonical output for English prompts. Maybe what I’m finding is that tokens the model is worse at generating are more likely to be non-canonical, and that rate drops with larger models and over time as training gets better.
Regarding “impossible behavior”, I just mean that the transcript makes it look like the toy model (2 layer, dim=16) did something that it can’t do, not that the capability is impossible in general. It was sized specifically to ensure that it couldn’t do one-step 3-digit-addition but could do it with non-canonical output.
I’m still in early phases of thinking about how to generate a more natural example of this. I was thinking a badly designed monitor could take a transcript (text) and tokenize it (canonically) and then use linear probes to decide if there’s concerning behavior. If the model generates non-canonical tokens, they would be hidden when the tokens are converted to a transcript. I’m not entirely sure how realistic of a threat vector this is, but it would be useful to know if a model can easily learn to evade this or not.
Hmm, makes sense, thanks for sharing! In this case it seems that small models indeed underfit on the canonical tokenization (due to lacking capacity to approximate the true distribution and oversmoothing possibly, in a sense of Morris et al., 2025). Especially given that the behavior is language-dependent and tokenizers are predominantly trained on English datasets.
Toy model is interesting. Does it mean that learning addition for inputs tokenized the way that they are “easier” to comprehend and infer a working algorithm for, make it easier for the model to generalize on? Such framing seems to make it sound trivial, yet it isn’t given that tokenization doesn’t work this way most of the time...
Probing experiment sounds great btw. Again, I don’t think I have an answer yet whether “non-canonicity” can be easily identifiable (linearly or not—some methods report to use KNN classifiers which is reasonable as there’re exponential number of ways non-canonicity can be imposed)
They toy model was designed around the idea that doing addition one digit at a time is much easier than multi-digit addition, since each digit you write is sort of like CoT (and this is especially easy if you do the addition backwards). I only showed that this is possible as a proof of concept though. The toy model is directly trained to have this behavior, and I’m not sure if it could learn it on its own. The fact that existing models with awful digit tokenization don’t seem to learn this on their own (as far as I know[1]?) is evidence against the theory.
Specifically, it was trained on a mix of input and output formats using 1 to 3 digit tokens (all using teacher forcing), and it successfully learned to do addition using 1-digit tokens for both input and output, and failed to learn the multi-digit token version. The number of parameters was specifically chosen after a sweep to ensure that this would happen (I used dim=16, but around dim=64 it starts to succeed at this task with 3-digit tokens).
This gave me an idea, and it seems like modern LMs don’t seem to output non-canonical digits. I’m going to check if RL post-training on GPT-2 can cause this to arise spontaneously though, since it has insane digit tokenization and sometimes outputs non-canonical digits by default.
Super cool! Non-canonical emission and associated hidden computations are right up our alley, great to hear you have been thinking about this as well. I hadn’t considered prompting intentionally to obtain NC sequences, this is cool. I really like your case study of addition here. I’ll check out the code!