While the Chain-of-Thought (CoT) for OpenAI models reasonably accurately reflects the model’s cognition, the CoT for Anthropic models does so to a substantially lesser extent. This may be due to “spillover” effects where reinforcement on outputs transfers to the CoT because Anthropic’s CoT is less distinct from the output
Could primary main cause instead be (as Tim Hua notes) that Anthropic has trained past models to have reward-model-pleasing chains of thought?
Yep, also seems possible, though I currently expect spillover is a larger effect. And there doesn’t seem to be a big Opus 4.5 vs Opus 4.6 gap implying this specific issue isn’t that likely to be the problem (insofar as I’m correct to guess that the CoT is less representative and this is due to training pressures). (As is obvious, I wrote this before the Mythos system card came out.)
Could primary main cause instead be (as Tim Hua notes) that Anthropic has trained past models to have reward-model-pleasing chains of thought?
Yep, also seems possible, though I currently expect spillover is a larger effect. And there doesn’t seem to be a big Opus 4.5 vs Opus 4.6 gap implying this specific issue isn’t that likely to be the problem (insofar as I’m correct to guess that the CoT is less representative and this is due to training pressures). (As is obvious, I wrote this before the Mythos system card came out.)