For non-canonical output dropping with size, I think we’re measuring different quantities (I measured per-token instead of per-generation), but I got a strong trend within families and a weaker one across them. This is kind of messy though since I got ~0% non-canonical output for English prompts. Maybe what I’m finding is that tokens the model is worse at generating are more likely to be non-canonical, and that rate drops with larger models and over time as training gets better.
Regarding “impossible behavior”, I just mean that the transcript makes it look like the toy model (2 layer, dim=16) did something that it can’t do, not that the capability is impossible in general. It was sized specifically to ensure that it couldn’t do one-step 3-digit-addition but could do it with non-canonical output.
I’m still in early phases of thinking about how to generate a more natural example of this. I was thinking a badly designed monitor could take a transcript (text) and tokenize it (canonically) and then use linear probes to decide if there’s concerning behavior. If the model generates non-canonical tokens, they would be hidden when the tokens are converted to a transcript. I’m not entirely sure how realistic of a threat vector this is, but it would be useful to know if a model can easily learn to evade this or not.
Hmm, makes sense, thanks for sharing! In this case it seems that small models indeed underfit on the canonical tokenization (due to lacking capacity to approximate the true distribution and oversmoothing possibly, in a sense of Morris et al., 2025). Especially given that the behavior is language-dependent and tokenizers are predominantly trained on English datasets.
Toy model is interesting. Does it mean that learning addition for inputs tokenized the way that they are “easier” to comprehend and infer a working algorithm for, make it easier for the model to generalize on? Such framing seems to make it sound trivial, yet it isn’t given that tokenization doesn’t work this way most of the time...
Probing experiment sounds great btw. Again, I don’t think I have an answer yet whether “non-canonicity” can be easily identifiable (linearly or not—some methods report to use KNN classifiers which is reasonable as there’re exponential number of ways non-canonicity can be imposed)
They toy model was designed around the idea that doing addition one digit at a time is much easier than multi-digit addition, since each digit you write is sort of like CoT (and this is especially easy if you do the addition backwards). I only showed that this is possible as a proof of concept though. The toy model is directly trained to have this behavior, and I’m not sure if it could learn it on its own. The fact that existing models with awful digit tokenization don’t seem to learn this on their own (as far as I know[1]?) is evidence against the theory.
Specifically, it was trained on a mix of input and output formats using 1 to 3 digit tokens (all using teacher forcing), and it successfully learned to do addition using 1-digit tokens for both input and output, and failed to learn the multi-digit token version. The number of parameters was specifically chosen after a sweep to ensure that this would happen (I used dim=16, but around dim=64 it starts to succeed at this task with 3-digit tokens).
This gave me an idea, and it seems like modern LMs don’t seem to output non-canonical digits. I’m going to check if RL post-training on GPT-2 can cause this to arise spontaneously though, since it has insane digit tokenization and sometimes outputs non-canonical digits by default.
For non-canonical output dropping with size, I think we’re measuring different quantities (I measured per-token instead of per-generation), but I got a strong trend within families and a weaker one across them. This is kind of messy though since I got ~0% non-canonical output for English prompts. Maybe what I’m finding is that tokens the model is worse at generating are more likely to be non-canonical, and that rate drops with larger models and over time as training gets better.
Regarding “impossible behavior”, I just mean that the transcript makes it look like the toy model (2 layer, dim=16) did something that it can’t do, not that the capability is impossible in general. It was sized specifically to ensure that it couldn’t do one-step 3-digit-addition but could do it with non-canonical output.
I’m still in early phases of thinking about how to generate a more natural example of this. I was thinking a badly designed monitor could take a transcript (text) and tokenize it (canonically) and then use linear probes to decide if there’s concerning behavior. If the model generates non-canonical tokens, they would be hidden when the tokens are converted to a transcript. I’m not entirely sure how realistic of a threat vector this is, but it would be useful to know if a model can easily learn to evade this or not.
Hmm, makes sense, thanks for sharing! In this case it seems that small models indeed underfit on the canonical tokenization (due to lacking capacity to approximate the true distribution and oversmoothing possibly, in a sense of Morris et al., 2025). Especially given that the behavior is language-dependent and tokenizers are predominantly trained on English datasets.
Toy model is interesting. Does it mean that learning addition for inputs tokenized the way that they are “easier” to comprehend and infer a working algorithm for, make it easier for the model to generalize on? Such framing seems to make it sound trivial, yet it isn’t given that tokenization doesn’t work this way most of the time...
Probing experiment sounds great btw. Again, I don’t think I have an answer yet whether “non-canonicity” can be easily identifiable (linearly or not—some methods report to use KNN classifiers which is reasonable as there’re exponential number of ways non-canonicity can be imposed)
They toy model was designed around the idea that doing addition one digit at a time is much easier than multi-digit addition, since each digit you write is sort of like CoT (and this is especially easy if you do the addition backwards). I only showed that this is possible as a proof of concept though. The toy model is directly trained to have this behavior, and I’m not sure if it could learn it on its own. The fact that existing models with awful digit tokenization don’t seem to learn this on their own (as far as I know[1]?) is evidence against the theory.
Specifically, it was trained on a mix of input and output formats using 1 to 3 digit tokens (all using teacher forcing), and it successfully learned to do addition using 1-digit tokens for both input and output, and failed to learn the multi-digit token version. The number of parameters was specifically chosen after a sweep to ensure that this would happen (I used dim=16, but around dim=64 it starts to succeed at this task with 3-digit tokens).
This gave me an idea, and it seems like modern LMs don’t seem to output non-canonical digits. I’m going to check if RL post-training on GPT-2 can cause this to arise spontaneously though, since it has insane digit tokenization and sometimes outputs non-canonical digits by default.