One part that seems confused to me, on chain-of-thought as an interpretability method:
Another limitation [of chain-of-thought] arises from restricted generality. Chains of thought will typically only serve as a propositional interpretibility for chain-of-thought systems: systems that use chains of thought for reasoning. For systems that do not themselves use chains of thought, any chains of thought that we generate will play no role in the system. Once chains of thought are unmoored from the original system in this way, it is even more unclear why they should reflect it. Of course we could try to find some way to train a non-chain-of-thought system to make accurate reports of its internal states along the way – but that is just the thought-logging problem all over again,and chains of thought will play no special role.
I have no idea what he’s trying to say here. Is he somehow totally unaware that you can use CoT with regular LLMs? That seems unlikely since he’s clearly read some of the relevant literature (eg he cites Turpin et al’s paper on CoT unfaithfulness). I don’t know how else to interpret it, though—maybe I’m just missing something?
Regular LLMs can use chain-of-thought reasoning. He is speaking about generating chains of thought for systems that don’t use them. E.g. AlphaGo, or diffusion models, or even an LLM in cases where it didn’t use CoT but produced the answer immediately.
As an example, you ask an LLM a question, and it answers it without using CoT. Then you ask it to explain how it arrived at its answer. It will generate something for you that looks like a chain of thought. But since it wasn’t literally using it while producing its original answer, this is just an after-the-fact rationalization. It is questionable whether such a post-hoc “chain of thought” reflects anything the model was actually doing internally when it originally came up with the answer. It could be pure confabulation.
Your first paragraph makes sense as an interpretation, which I discounted because the idea of something like AlphaGo doing CoT (or applying a CoT to it) seems so nonsensical, since it’s not at all a linguistic model.
I’m having more trouble seeing how to read what Chalmer says in the way your second paragraph suggests—eg ‘unmoored from the original system’ doesn’t seem like it’s talking about the same system generating an ad hoc explanation. It’s more like he’s talking about somehow taking a CoT generated by one model and applying it to another, although that also seems nonsensical.
If you want to understand why a model, any model, did something, you presumably want a verbal explanation of its reasoning, a chain of thought. E.g. why AlphaGo made its famous unexpected move 37. That’s not just true for language models.
One part that seems confused to me, on chain-of-thought as an interpretability method:
I have no idea what he’s trying to say here. Is he somehow totally unaware that you can use CoT with regular LLMs? That seems unlikely since he’s clearly read some of the relevant literature (eg he cites Turpin et al’s paper on CoT unfaithfulness). I don’t know how else to interpret it, though—maybe I’m just missing something?
Regular LLMs can use chain-of-thought reasoning. He is speaking about generating chains of thought for systems that don’t use them. E.g. AlphaGo, or diffusion models, or even an LLM in cases where it didn’t use CoT but produced the answer immediately.
As an example, you ask an LLM a question, and it answers it without using CoT. Then you ask it to explain how it arrived at its answer. It will generate something for you that looks like a chain of thought. But since it wasn’t literally using it while producing its original answer, this is just an after-the-fact rationalization. It is questionable whether such a post-hoc “chain of thought” reflects anything the model was actually doing internally when it originally came up with the answer. It could be pure confabulation.
Your first paragraph makes sense as an interpretation, which I discounted because the idea of something like AlphaGo doing CoT (or applying a CoT to it) seems so nonsensical, since it’s not at all a linguistic model.
I’m having more trouble seeing how to read what Chalmer says in the way your second paragraph suggests—eg ‘unmoored from the original system’ doesn’t seem like it’s talking about the same system generating an ad hoc explanation. It’s more like he’s talking about somehow taking a CoT generated by one model and applying it to another, although that also seems nonsensical.
If you want to understand why a model, any model, did something, you presumably want a verbal explanation of its reasoning, a chain of thought. E.g. why AlphaGo made its famous unexpected move 37. That’s not just true for language models.
Sure, I agree that would be useful.