In particular, does anyone have any view on if this is just a random meltdown or if this is actually what Fable’s CoT looks like? If the latter, perhaps that should substantially accelerate our timelines? (On the logic that reinforcement learning can accomplish more than was understood.)
Well, if RL “really worked”, yes it seems like it should almost certainly invent and reason in an alien language. But with all the models where we can actually see the CoT, it’s more or less in English. My interpretation (and I think the standard interpretation?) for that is that RL is hard and can’t really optimize too far from the parameters found in pretraining.
That step isn’t necessary for the argument. The point is just that you’re encouraging extra compression rather than incentivising the model to invent a language from scratch.
Well, if RL “really worked”, yes it seems like it should almost certainly invent and reason in an alien language.
What would be the advantage? Inventing new languages at least doesn’t seem to increase our reasoning ability compared to using existing ones. For example, it seems unlikely that people can reason better with any artificial languages like Esperanto or Interlingua (or Python, UML, etc) than with English. I would expect that if it were possible to artificially create a language that is substantially better for thinking than existing languages, we would have already created such a language.
Of course humans can also reason to a significant degree in latent space, without language, but latent reasoning (Neuralese) is different from reasoning in an artificial language (Thinkish).
I’m not sure whether I’d expect a whole new language with its own grammar. But with sufficiently capable RL, I do expect new words, as handles for new concepts the model came up with.
I guess the general argument is that the optimization space is very high-dimensional, and there’s no pressure to reason in English, so it just seems incredibly unlikely that the optimum would look like thinking aloud in English. I’m not sure how much I buy the human analogy; we have different capabilities and IMO humans don’t think in language to the same degree that current LLMs do. But certainly humans in specialized areas find it useful to invent lots of specialized terminology and conventions. If you sent people to an island and had them live for 10000 years with their society entirely devoted to chess, I’d conjecture that their language would evolve quite a bit. If RL could “really do RL” it should be able to create such evolutions for many domains, or (I suspect) cross-domain.
This seems out of line with what was said in the GRPO paper way back when. They had to alternate between RLVR and supervised learning to make sure the model still spoke English properly when it was done.
DeepSeek (which GRPO was invented for) still uses a normal English CoT to this day. (V4 Pro) They had to alternate with supervised learning because RL is just very very hard and GRPO is incredibly weak compared to supervised learning. They aren’t deliberately sacrificing performance by forcing an English CoT instead of a more useful or efficient neuralese. They are forced to do it because RL just doesn’t work that well.
I didn’t disagree-vote but I found your quick take confusing because the example you linked seemed on-trend with the general progression of leaked frontier CoTs (mostly OpenAI, though) I have seen over the past 1.5 years to me.
Thanks, so is the interpretation something like this?
2a. “this is what Fable’s CoT looks like, but it doesn’t follow that we should accelerate our timelines, because we already knew RL could produce this kind of behavior, so it should already be baked into your timelines if your timelines are good.”
I don’t mind disagree votes, I’m just interested to see what people think. I was very surprised to see that CoT, but I’m only really basing that on reading CoT from open models (Kimi, DeepSeek, etc.) that do sometimes have random meltdowns, but AFAIK never do something like this productively.
Has there been any discussion here of the “leaked Fable chain of thought”?
https://old.reddit.com/r/ClaudeAI/comments/1ul1396/fable_5_leaked_chainofthought_in_web_interface/
In particular, does anyone have any view on if this is just a random meltdown or if this is actually what Fable’s CoT looks like? If the latter, perhaps that should substantially accelerate our timelines? (On the logic that reinforcement learning can accomplish more than was understood.)
“On the logic that reinforcement learning can accomplish more than was understood.”
Could you explain this? It seemed obvious that RL results in difficult to parse CoT.
Well, if RL “really worked”, yes it seems like it should almost certainly invent and reason in an alien language. But with all the models where we can actually see the CoT, it’s more or less in English. My interpretation (and I think the standard interpretation?) for that is that RL is hard and can’t really optimize too far from the parameters found in pretraining.
“invent and reason in an alien language”
That step isn’t necessary for the argument. The point is just that you’re encouraging extra compression rather than incentivising the model to invent a language from scratch.
What would be the advantage? Inventing new languages at least doesn’t seem to increase our reasoning ability compared to using existing ones. For example, it seems unlikely that people can reason better with any artificial languages like Esperanto or Interlingua (or Python, UML, etc) than with English. I would expect that if it were possible to artificially create a language that is substantially better for thinking than existing languages, we would have already created such a language.
Of course humans can also reason to a significant degree in latent space, without language, but latent reasoning (Neuralese) is different from reasoning in an artificial language (Thinkish).
I’m not sure whether I’d expect a whole new language with its own grammar. But with sufficiently capable RL, I do expect new words, as handles for new concepts the model came up with.
I guess the general argument is that the optimization space is very high-dimensional, and there’s no pressure to reason in English, so it just seems incredibly unlikely that the optimum would look like thinking aloud in English. I’m not sure how much I buy the human analogy; we have different capabilities and IMO humans don’t think in language to the same degree that current LLMs do. But certainly humans in specialized areas find it useful to invent lots of specialized terminology and conventions. If you sent people to an island and had them live for 10000 years with their society entirely devoted to chess, I’d conjecture that their language would evolve quite a bit. If RL could “really do RL” it should be able to create such evolutions for many domains, or (I suspect) cross-domain.
This seems out of line with what was said in the GRPO paper way back when. They had to alternate between RLVR and supervised learning to make sure the model still spoke English properly when it was done.
DeepSeek (which GRPO was invented for) still uses a normal English CoT to this day. (V4 Pro) They had to alternate with supervised learning because RL is just very very hard and GRPO is incredibly weak compared to supervised learning. They aren’t deliberately sacrificing performance by forcing an English CoT instead of a more useful or efficient neuralese. They are forced to do it because RL just doesn’t work that well.
Anthropic themselves briefly discuss “illegible” CoT in the Mythos & Fable system card; the example they provide is on page 108 and has also been discussed on LW
Perfect, thanks, just what I was looking for.
This Twitter user is known for having an undisclosed method for accessing the CoT of OpenAI models. He sometimes posts snippets. Recent examples.
Not sure how to interpret the accuracy downvotes. :) Do they mean:
The explanation is something other than “random meltdown” or “this is what Fable’s CoT looks like”?
The explanation is “this is what Fable’s CoT looks like, but it doesn’t follow that we should accelerate our timelines”?
Something else?
I didn’t disagree-vote but I found your quick take confusing because the example you linked seemed on-trend with the general progression of leaked frontier CoTs (mostly OpenAI, though) I have seen over the past 1.5 years to me.
Thanks, so is the interpretation something like this?
2a. “this is what Fable’s CoT looks like, but it doesn’t follow that we should accelerate our timelines, because we already knew RL could produce this kind of behavior, so it should already be baked into your timelines if your timelines are good.”
I don’t mind disagree votes, I’m just interested to see what people think. I was very surprised to see that CoT, but I’m only really basing that on reading CoT from open models (Kimi, DeepSeek, etc.) that do sometimes have random meltdowns, but AFAIK never do something like this productively.
From hovering I can see there are just three accuracy votes (for a total of −11) so far, so someone apparently used strong downvotes.
I’m just curious if it’s common for chain of thought to have so many math quantifiers like \exists. Or am I reading that wrong?