Can we avoid incentivizing the AI to endlessly compress its chain of thoughts, by giving it tokens to represent long strings of English words which frequently appear in chains of thought?
The AI doesn’t want to switch to a compressed alien language, since it has to learn to both read and write a new token to represent a concept. But things like context window limitations force it to.
I suspect the append-heavy flows we see today are more driven by cost optimization (prompt caching makes inference much cheaper) rather than because feeding all past thoughts + outputs + tool calls + tool results + every version of every edited file is actually the most effective use of tokens.
Oops I shouldn’t have pointed to context window limitations. Probably the real reason the AI is incentivized to compress its chain of thought is because of its output token budget/effort level, not context window limitations (they can remember a million tokens and compact long chains of thought).
Can we avoid incentivizing the AI to endlessly compress its chain of thoughts, by giving it tokens to represent long strings of English words which frequently appear in chains of thought?
The AI doesn’t want to switch to a compressed alien language, since it has to learn to both read and write a new token to represent a concept. But things like context window limitations force it to.
I suspect the append-heavy flows we see today are more driven by cost optimization (prompt caching makes inference much cheaper) rather than because feeding all past thoughts + outputs + tool calls + tool results + every version of every edited file is actually the most effective use of tokens.
Oops I shouldn’t have pointed to context window limitations. Probably the real reason the AI is incentivized to compress its chain of thought is because of its output token budget/effort level, not context window limitations (they can remember a million tokens and compact long chains of thought).