Thanks for the investigation and sharing! The explorations on the reasoning models is seems new and a good extension.
Regarding the non-reasoning results, one might speculate that the multi-token prediction part of the architecture[1] could influence some of the anomalous token behavior. Tokens that are almost always bigram continuations (eg, “eredWriter”, “reeNode”, “VERTISEMENT”) likely almost always are predicted by the two-ahead predictor. Thus the model might get further confused when it must try to generate these tokens via the next token predictor. We might also speculate there are ways two-ahead predictors could increase the risk of ending up in repeating basins.
It would be interested to explore this more quantitatively on how/if the deepseek architecture differs. Your preliminary work is valuable, but I don’t actually gives evidence this model has more or less anomalous token behavior than say GPT or LLaMa. Also, if there are differences, it could be architectural, or just quirks of the training.
Thanks for the investigation and sharing! The explorations on the reasoning models is seems new and a good extension.
Regarding the non-reasoning results, one might speculate that the multi-token prediction part of the architecture[1] could influence some of the anomalous token behavior. Tokens that are almost always bigram continuations (eg, “eredWriter”, “reeNode”, “VERTISEMENT”) likely almost always are predicted by the two-ahead predictor. Thus the model might get further confused when it must try to generate these tokens via the next token predictor. We might also speculate there are ways two-ahead predictors could increase the risk of ending up in repeating basins.
It would be interested to explore this more quantitatively on how/if the deepseek architecture differs. Your preliminary work is valuable, but I don’t actually gives evidence this model has more or less anomalous token behavior than say GPT or LLaMa. Also, if there are differences, it could be architectural, or just quirks of the training.
Ege Edril has a nice short summary of this architecture component for someone unfamiliar