plex comments on The Dark Arts of Tokenization or: How I learned to start worrying and love LLMs’ undecoded outputs

[plex]

Right, I think that’s the hypothesis I was asking whether you had when I said

unless you mean the randomness was in an underlying valance factor?

If so, yeah, this is compatible. I’d still put notably higher odds on the original thing I suggested, but this is the other main hypothesis.