Nathan Helm-Burger comments on Detecting out of distribution text with surprisal and entropy

[Nathan Helm-Burger]

Training with ‘patches’ instead of subword tokens: https://​​arxiv.org/​​html/​​2407.12665v2