One key ingredient for metacognition is “stopping to think” at appropriate points.
There are certain mental tics one observes in CoT from reasoning models:
”But wait! …” ″Perfect! …”
A human might not even waste subvocalized syllables on these — we might jump straight to the … part. But an LLM has to emit a token to make and record a decision so it can attend to it later. So you see these characteristic tics.
That’s right! The word “wait” is reportedly very common in reasoning CoT. I think it’s playing the role of the basal ganglia “stop” mechanism I mentioned.
Beyond using “wait” properly to reconsider, it looks to me like LLMs are still pretty crappy at integrating multiple lines of thought. They’re too prone to just accept the results of the second one, even if they’re worse. It does look like more-is-better on average, but adding the additional metacognitive skills to more carefully integrate conflicting approaches and answers seems like it would help a lot.
I suspect that the LLMs’ problems with metacognition are due to the nature of LLMs and CoTs.
The LLM, unlike the humans, doesn’t change when doing a task. Instead, selected tokens from things like the prompt, the CoT, external documents found or created by the model are stuffed into the same mechanism ejecting the next token of the CoT, output, request, etc. In order to “more carefully integrate conflicting approaches” pursued in different parts of the CoT, the LLM would have to select the tokens from those parts. Were the LLM to change when doing a task (e.g. to be finetuned on the fly to produce the next token) during the entire training[1], it would have a chance to remember something from old approaches deeper. SOTA LLMs are not raised this way.
Unlike the discrete nature of the CoTs, neuralese is both continuous and has a higher bandwidth. This could, in theory, allow the LLM to, say, accumulate suspicion and act upon it rather than course-correct after it becomes clear that the LLM made a mistake.
And the Spiral Bench, which had KimiK2 become the least sycophantic model, less sycophantic even than GPT-5.2. However, KimiK2 was also the model roleplaying as the user. Tim Hua’s experiment had Grok roleplay as the user.
My impression is that studies have shown that, at least for earlier rounds of reasoning models where the total computation invested in reasoning training was fairly small compared to pretraining, reasoning training was mostly up-or-down-regulating skills, many of the metacognitive, that the base model already had, to optimize them happening at the right times and frequencies.
With sufficiently large amounts of reasoning training, one would expect metacognitive skills to improve. But already having the skill present in the base model from us is still going to be very helpful, I strongly suspect.
There are certain mental tics one observes in CoT from reasoning models:
”But wait! …”
″Perfect! …”
A human might not even waste subvocalized syllables on these — we might jump straight to the … part. But an LLM has to emit a token to make and record a decision so it can attend to it later. So you see these characteristic tics.
That’s right! The word “wait” is reportedly very common in reasoning CoT. I think it’s playing the role of the basal ganglia “stop” mechanism I mentioned.
Beyond using “wait” properly to reconsider, it looks to me like LLMs are still pretty crappy at integrating multiple lines of thought. They’re too prone to just accept the results of the second one, even if they’re worse. It does look like more-is-better on average, but adding the additional metacognitive skills to more carefully integrate conflicting approaches and answers seems like it would help a lot.
I suspect that the LLMs’ problems with metacognition are due to the nature of LLMs and CoTs.
The LLM, unlike the humans, doesn’t change when doing a task. Instead, selected tokens from things like the prompt, the CoT, external documents found or created by the model are stuffed into the same mechanism ejecting the next token of the CoT, output, request, etc. In order to “more carefully integrate conflicting approaches” pursued in different parts of the CoT, the LLM would have to select the tokens from those parts. Were the LLM to change when doing a task (e.g. to be finetuned on the fly to produce the next token) during the entire training[1], it would have a chance to remember something from old approaches deeper. SOTA LLMs are not raised this way.
Unlike the discrete nature of the CoTs, neuralese is both continuous and has a higher bandwidth. This could, in theory, allow the LLM to, say, accumulate suspicion and act upon it rather than course-correct after it becomes clear that the LLM made a mistake.
The final problem was the LLMs being overly sycophantic. The results of Tim Hua’s experiment[2] made me strongly suspect that this problem was solved by KimiK2-like training for satisfying self-critique instead of satisfying human critics who like being praised.
Attempts to finetune the LLM by using far less compute cause the LLM’s performance to drop.
And the Spiral Bench, which had KimiK2 become the least sycophantic model, less sycophantic even than GPT-5.2. However, KimiK2 was also the model roleplaying as the user. Tim Hua’s experiment had Grok roleplay as the user.
My impression is that studies have shown that, at least for earlier rounds of reasoning models where the total computation invested in reasoning training was fairly small compared to pretraining, reasoning training was mostly up-or-down-regulating skills, many of the metacognitive, that the base model already had, to optimize them happening at the right times and frequencies.
With sufficiently large amounts of reasoning training, one would expect metacognitive skills to improve. But already having the skill present in the base model from us is still going to be very helpful, I strongly suspect.