Try speaking aloud for precisely 269 words. You’re not allowed to count or recite poetry — you have to do this while actually extemporizing something interesting to say.
Now bear in mind that an LLM doesn’t output letters or words, it uses tokens. In order to count words, an LLM has to memorize which tokens contain spaces and which don’t. So for it the task is comparable to asking a human to speak until they’ve said 269 words that begin with the letter T.
Try speaking aloud for precisely 269 words. You’re not allowed to count or recite poetry — you have to do this while actually extemporizing something interesting to say.
Now bear in mind that an LLM doesn’t output letters or words, it uses tokens. In order to count words, an LLM has to memorize which tokens contain spaces and which don’t. So for it the task is comparable to asking a human to speak until they’ve said 269 words that begin with the letter T.