I appreciate the reference, although I found this article + discussion pretty underwhelming; it’s part of what’s motivating my question.
For instance, not all forms of unintelligibility in CoT’s are necessarily evidence of a drive-to-compression. But the article takes for granted that the weirdness we see in chains-of-thought are evidence towards this; it views various forms of weird text that I’d see as evidence for screwed up training systems or spandrells of the training process and just assumes they are “thinking” driven into non-human-legible vocabulary. The guy didn’t particularly consider other hypotheses for what he was seeing.
And similarly he discusses “redundancy” in human languages, and immediately assumes machines would want it to go away, while not… thinking of why it’s there, and whether it would stick around for machines potentially.
This isn’t anything like a full refutation of him, tbc, I’m just giving my impression of it at a high level. By my takeaway is that if this is the best discussion than I don’t think anyone’s actually tried to work out the reasoning around this carefully, even if neuralese is actually inevitable.
I appreciate the reference, although I found this article + discussion pretty underwhelming; it’s part of what’s motivating my question.
For instance, not all forms of unintelligibility in CoT’s are necessarily evidence of a drive-to-compression. But the article takes for granted that the weirdness we see in chains-of-thought are evidence towards this; it views various forms of weird text that I’d see as evidence for screwed up training systems or spandrells of the training process and just assumes they are “thinking” driven into non-human-legible vocabulary. The guy didn’t particularly consider other hypotheses for what he was seeing.
And similarly he discusses “redundancy” in human languages, and immediately assumes machines would want it to go away, while not… thinking of why it’s there, and whether it would stick around for machines potentially.
This isn’t anything like a full refutation of him, tbc, I’m just giving my impression of it at a high level. By my takeaway is that if this is the best discussion than I don’t think anyone’s actually tried to work out the reasoning around this carefully, even if neuralese is actually inevitable.