Do you refer to the way new tokens (that might end up underfitted) are introduced into the tokenizers during post-training or any other stages, when you are talking about “auto-refactoring”
yes
which kind of “token libraries” are you refering to?
I mentioned math and code (here I include structured markup) - both of which markedly enhanced performance in corresponding subject matter areas when incorporated (although you seem to be working at the tokenizer level rather than the corpus level). I believe CoT reasoning traces themselves are another example. Finance data seems to be a counterexample (to even domain enhancement, much less more broadly). It’ll be interesting to see what materials/hardware tokenizations look like and the effect they have.
I might be unfamiliar with some line of research
I’m almost certainly using non-standard terminology, my apologies.
yes
I mentioned math and code (here I include structured markup) - both of which markedly enhanced performance in corresponding subject matter areas when incorporated (although you seem to be working at the tokenizer level rather than the corpus level). I believe CoT reasoning traces themselves are another example. Finance data seems to be a counterexample (to even domain enhancement, much less more broadly). It’ll be interesting to see what materials/hardware tokenizations look like and the effect they have.
I’m almost certainly using non-standard terminology, my apologies.