They toy model was designed around the idea that doing addition one digit at a time is much easier than multi-digit addition, since each digit you write is sort of like CoT (and this is especially easy if you do the addition backwards). I only showed that this is possible as a proof of concept though. The toy model is directly trained to have this behavior, and I’m not sure if it could learn it on its own. The fact that existing models with awful digit tokenization don’t seem to learn this on their own (as far as I know[1]?) is evidence against the theory.
Specifically, it was trained on a mix of input and output formats using 1 to 3 digit tokens (all using teacher forcing), and it successfully learned to do addition using 1-digit tokens for both input and output, and failed to learn the multi-digit token version. The number of parameters was specifically chosen after a sweep to ensure that this would happen (I used dim=16, but around dim=64 it starts to succeed at this task with 3-digit tokens).
This gave me an idea, and it seems like modern LMs don’t seem to output non-canonical digits. I’m going to check if RL post-training on GPT-2 can cause this to arise spontaneously though, since it has insane digit tokenization and sometimes outputs non-canonical digits by default.
They toy model was designed around the idea that doing addition one digit at a time is much easier than multi-digit addition, since each digit you write is sort of like CoT (and this is especially easy if you do the addition backwards). I only showed that this is possible as a proof of concept though. The toy model is directly trained to have this behavior, and I’m not sure if it could learn it on its own. The fact that existing models with awful digit tokenization don’t seem to learn this on their own (as far as I know[1]?) is evidence against the theory.
Specifically, it was trained on a mix of input and output formats using 1 to 3 digit tokens (all using teacher forcing), and it successfully learned to do addition using 1-digit tokens for both input and output, and failed to learn the multi-digit token version. The number of parameters was specifically chosen after a sweep to ensure that this would happen (I used dim=16, but around dim=64 it starts to succeed at this task with 3-digit tokens).
This gave me an idea, and it seems like modern LMs don’t seem to output non-canonical digits. I’m going to check if RL post-training on GPT-2 can cause this to arise spontaneously though, since it has insane digit tokenization and sometimes outputs non-canonical digits by default.