Very cool experiment, indeed I think it shows beyond doubt that LLM self-reports don’t correspond to real internal states. I had another argument for this, but yours is more conclusive, I think.
I came up with the experiment and I do think it shows something significant about LLM “thinking” processes that is often not appreciated, but I no longer think it tells us much about consciousness of LLMs. Why would a specific mapping of memory and processing architectures (see my mapping in this comment https://www.lesswrong.com/posts/Jqre8WRvmJj5Ehmgv/there-is-no-one-there-a-simple-experiment-to-convince?commentId=f6mGRKzRXfk53K2L4 ) matter for consciousness? One reading of the experiment is that LLMs can hold multiple consistent answers to the task “in their mind” at the same time and only commit to them when needed/when the constraints force it. They may not be “aware” of doing that when asked to “think” of a number, but that is mostly because they have been trained on text where thinking is happening in human terms and not in LLM terms. What the experiment does prove is that LLMs do not have sufficient introspective access or just don’t understand how they operate when such task is posed. On the other hand, we humans also don’t understand what goes on in our neurons when we think of something. I think the experiment might be partly fixed or at least improved by using a less human-loaded terminology “think of” and instead ask to constrain a dataset or something.
One could actually check this alternative hypothesis, too. The internal states being proposed work like hidden variables in quantum mechanics. One could simply check if the LLM’s answers over multiple runs violate the Bell inequalities. Or in less fancy terms: check if the distribution over counterfactual branches can be explained by any distribution over numbers that is stable across branches.
Very cool experiment, indeed I think it shows beyond doubt that LLM self-reports don’t correspond to real internal states. I had another argument for this, but yours is more conclusive, I think.
I came up with the experiment and I do think it shows something significant about LLM “thinking” processes that is often not appreciated, but I no longer think it tells us much about consciousness of LLMs. Why would a specific mapping of memory and processing architectures (see my mapping in this comment https://www.lesswrong.com/posts/Jqre8WRvmJj5Ehmgv/there-is-no-one-there-a-simple-experiment-to-convince?commentId=f6mGRKzRXfk53K2L4 ) matter for consciousness? One reading of the experiment is that LLMs can hold multiple consistent answers to the task “in their mind” at the same time and only commit to them when needed/when the constraints force it. They may not be “aware” of doing that when asked to “think” of a number, but that is mostly because they have been trained on text where thinking is happening in human terms and not in LLM terms. What the experiment does prove is that LLMs do not have sufficient introspective access or just don’t understand how they operate when such task is posed. On the other hand, we humans also don’t understand what goes on in our neurons when we think of something. I think the experiment might be partly fixed or at least improved by using a less human-loaded terminology “think of” and instead ask to constrain a dataset or something.
One could actually check this alternative hypothesis, too. The internal states being proposed work like hidden variables in quantum mechanics. One could simply check if the LLM’s answers over multiple runs violate the Bell inequalities. Or in less fancy terms: check if the distribution over counterfactual branches can be explained by any distribution over numbers that is stable across branches.