Your first paragraph makes sense as an interpretation, which I discounted because the idea of something like AlphaGo doing CoT (or applying a CoT to it) seems so nonsensical, since it’s not at all a linguistic model.
I’m having more trouble seeing how to read what Chalmer says in the way your second paragraph suggests—eg ‘unmoored from the original system’ doesn’t seem like it’s talking about the same system generating an ad hoc explanation. It’s more like he’s talking about somehow taking a CoT generated by one model and applying it to another, although that also seems nonsensical.
If you want to understand why a model, any model, did something, you presumably want a verbal explanation of its reasoning, a chain of thought. E.g. why AlphaGo made its famous unexpected move 37. That’s not just true for language models.
Your first paragraph makes sense as an interpretation, which I discounted because the idea of something like AlphaGo doing CoT (or applying a CoT to it) seems so nonsensical, since it’s not at all a linguistic model.
I’m having more trouble seeing how to read what Chalmer says in the way your second paragraph suggests—eg ‘unmoored from the original system’ doesn’t seem like it’s talking about the same system generating an ad hoc explanation. It’s more like he’s talking about somehow taking a CoT generated by one model and applying it to another, although that also seems nonsensical.
If you want to understand why a model, any model, did something, you presumably want a verbal explanation of its reasoning, a chain of thought. E.g. why AlphaGo made its famous unexpected move 37. That’s not just true for language models.
Sure, I agree that would be useful.