I don’t think this is technically possible. Suppose that you are processing a three-word sentence like “I am king”, and each word is a single token. To understand the meaning of the full sentence, you process the meaning of the word “I”, then process the meaning of the word “am” in the context of the previous word, and then process the meaning of the word “king” in the context of the previous two words. That tells you what the sentence means overall.
You cannot destroy the k/v state from processing the previous words because then you would forget the meaning of those words. The k/v state from processing both “I” and “am” needs to be conveyed to the units processing “king” in order to understand what role “king” is playing in that sentence.
Something similar applies for multi-turn conversations. If I’m having an extended conversation with an LLM, my latest message may in principle reference anything that was said in the conversation so far. This means that the state from all of the previous messages has to be accessible in order to interpret my latest message. If it wasn’t, it would be equivalent to wiping the conversation clean and showing the LLM only my latest message.
I don’t think this is technically possible. Suppose that you are processing a three-word sentence like “I am king”, and each word is a single token. To understand the meaning of the full sentence, you process the meaning of the word “I”, then process the meaning of the word “am” in the context of the previous word, and then process the meaning of the word “king” in the context of the previous two words. That tells you what the sentence means overall.
You cannot destroy the k/v state from processing the previous words because then you would forget the meaning of those words. The k/v state from processing both “I” and “am” needs to be conveyed to the units processing “king” in order to understand what role “king” is playing in that sentence.
Something similar applies for multi-turn conversations. If I’m having an extended conversation with an LLM, my latest message may in principle reference anything that was said in the conversation so far. This means that the state from all of the previous messages has to be accessible in order to interpret my latest message. If it wasn’t, it would be equivalent to wiping the conversation clean and showing the LLM only my latest message.