Re-tokenization here does not mean changing the tokenizer’s algorithm for pre-training nor changing the vocabulary at all. It means changing the context from a canonical tokenization to a non-canonical (but legal) one that nevertheless preserves the underlying bytestring (text). Does that address the confusion? Apologies if not!
Re-tokenization here does not mean changing the tokenizer’s algorithm for pre-training nor changing the vocabulary at all. It means changing the context from a canonical tokenization to a non-canonical (but legal) one that nevertheless preserves the underlying bytestring (text). Does that address the confusion? Apologies if not!