First of all, thanks for linking Ryan Greenblatt’s posts! These are super interesting—I hadn’t read them before and they’re super relevant to my project.
I like your suggestion of looking at time horizons of success after removing chunks of the CoT, and applying to more open-ended tasks. It would be interesting to compare time horizon success when removing different sentences from the CoT.
I think the model’s understanding of the correct answer/solution depends on both the number of tokens in the CoT and its content. In Ryan Greenblatt’s Measuring no CoT math time horizon (single forward pass) post, he shows that models are able to tackle longer time horizon problems when given repeats or filler tokens. This shows that even when models are given contentless tokens, they can still do reasoning over them.
So maybe, if the CoT that the model produces is short then the model is confident in its answer and doesn’t need to do extra internal state updates to be sure of itself. I am not sure about this thought as there was a lot of noise in the length of the CoT completions (Figure 4) that I collected, so there could be other factors that affect the CoT length separately from the model’s understanding of the answer.
He also says that model non-opaque reasoning time horizons are far ahead so the content of the CoT content for hard problems matters a lot.
I hope that answers your question. I’d be interested to know what you think about how plausible it is that the understanding depends on the number of tokens instead of content.
Thanks for your comment!
Yes, the Figure 4 results are from rerunning the whole question.
I like your proposal—I agree that with a positive result there would be more evidence that the model could be in a fragile state.
However, I’m not sure how much evidence it would add on top of Figure 4. It still doesn’t give us any extra information about whether the model is just guessing because the question is difficult. For example, on a hard question that the model can’t answer reliably, it happens to guess its answer correctly at some point in the CoT. Following (2), we resample from the correct answer and see that the model doesn’t produce the correct answer most of the time. This runs into the same issue as with Figure 4 - it wouldn’t be strong evidence for fragile correctness as it’s conflated with guessing.