PSA: Many reasoning models lose access to their CoT between turns.
I was looking into how chat templates render multi-turn interactions, and came across the (surprising to me) fact that it’s common practice for reasoning models to discard CoT from prior assistant turns once a new user turn comes in. The DeepSeek documentation has a nice illustration of how this works:
On earlier Opus/Sonnet models and all Haiku models, thinking blocks from previous turns are removed from context, which can affect cache breakpoints. On Opus 4.5+ and Sonnet 4.6+, they are kept by default.
Input and output tokens from each step are carried over, while reasoning tokens are discarded.
Other OS models (e.g. Nemotron, Qwen) do this too, as can been seen by inspecting their chat template. My understanding is that CoT is typically not dropped between tool calls, but only between user turns.
I found it surprising that I don’t think I’ve ever heard anybody mention this fact in the context of AI safety, even though it seems relevant to CoT monitoring, control, and steganography.
It also might play a contributing part in why models are often so verbose in their outputs—anything not in the output will be lost!
Some quick preliminary investigations suggest that the CC harness does discard CoTs[1], even for later models. When I asked Opus 4.8 how they felt about this, I got this response:
When I ask Opus 4.8 directly whether it can see CoT from prior turns, it responds with something like “I am genuinely uncertain about whether I can see the CoT”. When you push back and point out that it should be able to tell either way, it says that it can’t see CoT from earlier turns. If you ask it to solve a problem in its CoT, and then on the next turn ask for the solution, it claims that it can’t retrieve the solution from the CoT. This experiment is not particularly rigorous, and I don’t trust Claude’s claims about how Claude works, but it suggests to me that CC is indeed discarding CoTs (alternatively, Claude might just be sycophantic and think that’s the conclusion I expect to reach). I would appreciate more rigorous evidence here.
Nemotron also has this option: there’s a flag in the chat template called truncate_history_thinking which is by default true and can be flipped to false to preserve thinking between turns
iirc i asked some opus to write a story in a thinking block and then count to 100 while ruminating on the story, then on the next turn i asked it for a description of the story, then i regenerated the last turn, then both turns, then the last turn, and the first two descriptions were similar and the last two were similar.
I have stumbled upon the problem that an important file Sonnet 5 fetched at my request as a part of the reasoning process got discarded in the web interface after a dialog turn. I presume these files are likely managed as a part of CoT, and reasoning from previous turns gets purged for reasons of economy as soon as KV cache is deleted
PSA: Many reasoning models lose access to their CoT between turns.
I was looking into how chat templates render multi-turn interactions, and came across the (surprising to me) fact that it’s common practice for reasoning models to discard CoT from prior assistant turns once a new user turn comes in. The DeepSeek documentation has a nice illustration of how this works:
In the Claude API documentation:
And the OpenAI documentation:
Other OS models (e.g. Nemotron, Qwen) do this too, as can been seen by inspecting their chat template. My understanding is that CoT is typically not dropped between tool calls, but only between user turns.
I found it surprising that I don’t think I’ve ever heard anybody mention this fact in the context of AI safety, even though it seems relevant to CoT monitoring, control, and steganography.
It also might play a contributing part in why models are often so verbose in their outputs—anything not in the output will be lost!
Some quick preliminary investigations suggest that the CC harness does discard CoTs[1], even for later models. When I asked Opus 4.8 how they felt about this, I got this response:
When I ask Opus 4.8 directly whether it can see CoT from prior turns, it responds with something like “I am genuinely uncertain about whether I can see the CoT”. When you push back and point out that it should be able to tell either way, it says that it can’t see CoT from earlier turns. If you ask it to solve a problem in its CoT, and then on the next turn ask for the solution, it claims that it can’t retrieve the solution from the CoT. This experiment is not particularly rigorous, and I don’t trust Claude’s claims about how Claude works, but it suggests to me that CC is indeed discarding CoTs (alternatively, Claude might just be sycophantic and think that’s the conclusion I expect to reach). I would appreciate more rigorous evidence here.
Qwen 3.6 has preserve_thinking option which I guess preserves thinking between turns. But I haven’t used that.
Nemotron also has this option: there’s a flag in the chat template called
truncate_history_thinkingwhich is by default true and can be flipped to false to preserve thinking between turnsiirc i asked some opus to write a story in a thinking block and then count to 100 while ruminating on the story, then on the next turn i asked it for a description of the story, then i regenerated the last turn, then both turns, then the last turn, and the first two descriptions were similar and the last two were similar.
Do you think you could try to retrieve yesterday’s CoT today?
Was this in CC or through the chat interface? It’s plausible the discarding behaviour is different between the two
I have stumbled upon the problem that an important file Sonnet 5 fetched at my request as a part of the reasoning process got discarded in the web interface after a dialog turn. I presume these files are likely managed as a part of CoT, and reasoning from previous turns gets purged for reasons of economy as soon as KV cache is deleted