[Question] Do LLMs Implement NLP Algorithms for Better Next Token Predictions?

Do you think that base LLMs implement forms of “meta” algorithms like TF-IDF to predict the next token better?

Intuitively, it would be pretty smart to proceed that way because a text tends to be coherent in terms of style and in terms of vocabulary, so it would be very useful for an LLM to implement internally some NLP algorithms that help a ton in guessing the next words.

It’s a question I’m interested in because if there’s any “meta” algorithm implemented internally whose efficiency depends on the training setup the LLM is in, it increases the chances that situational awareness would arise.

No comments.