Good summary of some pros and cons. Neutral lists of arguments both for and against something that don’t try to come to a conclusion are rationally virtuous, so kudos.
An addition point con-neuralese—it’s a mix of 1, 3, 7, but you didn’t mention it: I think CoT might give you a capability lever as well as an alignment lever.
Specifically, a CoT (or a stream of neuralese) could be legible to the model (or not legible). This may enable some process like this.
Suppose the process by which you arrived at some particular answer could be something you can survey, understand, and remark upon; one might ask oneself “how could I have done this reasoning faster” if you got the answer correct, or “hrm what was the root cause of the mistake I made here?” if you did not. An LLM can obviously do this with a legible CoT. It’s unclear or dubious that an AI could do these with neuralese.
Granted that you can survey, understand, or remark upon a CoT, this opens up a chance to improve capabilities by (2.1) having the LLM edit the CoT, inserting reflections like “Ah I made a mistake” or “Let’s enumerate some different possibilities...” (think all the kind of stuff @raemon likes to do), (2.2) running forward the CoT with the inserted reflections as prefill (2.3) training on the altered CoT iff the model gets more improvement. Lots of things you can do in (2.3)
In theory this should permit much richer and more reflective alterations to the CoT than “dumb” policy gradient updates.
The above is a rather specific proposal, and so has many burdensome details that are likely to be wrong, but I think something like this is not entirely unlikely.
From another angle—from one of the links you have above has Nostalgebraist talk about how from the users perspective, a legiible CoT is a capability enhancement:
In short: from an end user’s perspective, CoT visibility is a capabilities improvement.
I ended up just switching to 3.7 Sonnet for the task discussed above – not because it was “smarter” as a model in any way I knew about, but simply because the associated API made it so much easier to construct prompts that would effectively leverage its intelligence for my purposes.
This strikes me as a very encouraging sign for the CoT-monitoring alignment story.
But from my angle, this is also a capability bonus—because it allows the model to look at itself, etc?
I’m not sure about this of course, but I thought it worth adding.
Thanks, this is a good point and definitely worth adding to the list!
Assuming that this is useful for capabilities, I’d imagine that neuralese models would also be able to eventually develop mechanisms to reflect on their own reasoning processes. The kind of introspection that Anthropic recently studied might be seen as one precursor capability for this and as evidence that this is incentivized by the training process, even if the model also has the ability to reflect on the CoT. But of course, the ability to reflect on internal activations in a deep way might be much harder to develop than the ability to do this with a legible CoT.
Good summary of some pros and cons. Neutral lists of arguments both for and against something that don’t try to come to a conclusion are rationally virtuous, so kudos.
An addition point con-neuralese—it’s a mix of 1, 3, 7, but you didn’t mention it: I think CoT might give you a capability lever as well as an alignment lever.
Specifically, a CoT (or a stream of neuralese) could be legible to the model (or not legible). This may enable some process like this.
Suppose the process by which you arrived at some particular answer could be something you can survey, understand, and remark upon; one might ask oneself “how could I have done this reasoning faster” if you got the answer correct, or “hrm what was the root cause of the mistake I made here?” if you did not. An LLM can obviously do this with a legible CoT. It’s unclear or dubious that an AI could do these with neuralese.
Granted that you can survey, understand, or remark upon a CoT, this opens up a chance to improve capabilities by (2.1) having the LLM edit the CoT, inserting reflections like “Ah I made a mistake” or “Let’s enumerate some different possibilities...” (think all the kind of stuff @raemon likes to do), (2.2) running forward the CoT with the inserted reflections as prefill (2.3) training on the altered CoT iff the model gets more improvement. Lots of things you can do in (2.3)
In theory this should permit much richer and more reflective alterations to the CoT than “dumb” policy gradient updates.
The above is a rather specific proposal, and so has many burdensome details that are likely to be wrong, but I think something like this is not entirely unlikely.
From another angle—from one of the links you have above has Nostalgebraist talk about how from the users perspective, a legiible CoT is a capability enhancement:
But from my angle, this is also a capability bonus—because it allows the model to look at itself, etc?
I’m not sure about this of course, but I thought it worth adding.
Thanks, this is a good point and definitely worth adding to the list!
Assuming that this is useful for capabilities, I’d imagine that neuralese models would also be able to eventually develop mechanisms to reflect on their own reasoning processes. The kind of introspection that Anthropic recently studied might be seen as one precursor capability for this and as evidence that this is incentivized by the training process, even if the model also has the ability to reflect on the CoT. But of course, the ability to reflect on internal activations in a deep way might be much harder to develop than the ability to do this with a legible CoT.