Seems to me like at least a point in favor of “stochastic parrots” over “builds a quality world model” for the language reasoning models.
Also wondering if their findings could be used to the advantage of safety/security somehow. E.g. if these models are more dependent on imitating examples than we relaized, then it might also be more effective than we previously thought to purge training data of the types of knowledge and reasoning that we don’t want them to have (e.g. knowledge of dangerous weapons development, scheming, etc.)
I should have mentioned the above thoughts are a low-confidence take. I was mostly just trying to get the ball rolling on discussion because I couldn’t find any discussion of this paper on LessWrong yet, which really surprised me because I saw the paper had been shared thousands of times on LinkedIn already.
Thoughts on “The Ilusion of Thinking” paper that came out of Apple recently?
https://ml-site.cdn-apple.com/papers/the-illusion-of-thinking.pdf
Seems to me like at least a point in favor of “stochastic parrots” over “builds a quality world model” for the language reasoning models.
Also wondering if their findings could be used to the advantage of safety/security somehow. E.g. if these models are more dependent on imitating examples than we relaized, then it might also be more effective than we previously thought to purge training data of the types of knowledge and reasoning that we don’t want them to have (e.g. knowledge of dangerous weapons development, scheming, etc.)
I should have mentioned the above thoughts are a low-confidence take. I was mostly just trying to get the ball rolling on discussion because I couldn’t find any discussion of this paper on LessWrong yet, which really surprised me because I saw the paper had been shared thousands of times on LinkedIn already.
Starting to be some discussion on LW now, e.g.
https://www.lesswrong.com/posts/5uw26uDdFbFQgKzih/beware-general-claims-about-generalizable-reasoning
https://www.lesswrong.com/posts/tnc7YZdfGXbhoxkwj/give-me-a-reason-ing-model