Do you refer to the way new tokens (that might end up underfitted) are introduced into the tokenizers during post-training or any other stages, when you are talking about “auto-refactoring” or is it something else? And which kind of “token libraries” are you refering to?
I might be unfamiliar with some line of research, so I’m curious to find out
Do you refer to the way new tokens (that might end up underfitted) are introduced into the tokenizers during post-training or any other stages, when you are talking about “auto-refactoring”
yes
which kind of “token libraries” are you refering to?
I mentioned math and code (here I include structured markup) - both of which markedly enhanced performance in corresponding subject matter areas when incorporated (although you seem to be working at the tokenizer level rather than the corpus level). I believe CoT reasoning traces themselves are another example. Finance data seems to be a counterexample (to even domain enhancement, much less more broadly). It’ll be interesting to see what materials/hardware tokenizations look like and the effect they have.
I might be unfamiliar with some line of research
I’m almost certainly using non-standard terminology, my apologies.
Do you refer to the way new tokens (that might end up underfitted) are introduced into the tokenizers during post-training or any other stages, when you are talking about “auto-refactoring” or is it something else? And which kind of “token libraries” are you refering to?
I might be unfamiliar with some line of research, so I’m curious to find out
yes
I mentioned math and code (here I include structured markup) - both of which markedly enhanced performance in corresponding subject matter areas when incorporated (although you seem to be working at the tokenizer level rather than the corpus level). I believe CoT reasoning traces themselves are another example. Finance data seems to be a counterexample (to even domain enhancement, much less more broadly). It’ll be interesting to see what materials/hardware tokenizations look like and the effect they have.
I’m almost certainly using non-standard terminology, my apologies.