I think that this is a really important topic to think about, and we need more people thinking carefully about it, but I have a few points of disagreement:
Consciousness and introspective ability are quite different things. and the presence or absence of introspective ability doesn’t tell us much about whether LLMs are conscious (although, as you point out, the experiment you do here does suggest that their internal states are strange).
It’s important to recall that the output of an LLM isn’t a token; it’s a distribution over token probabilities. The output we see is just the result of following one such path. As a mental model, I like thinking of it as a dialogue tree in a computer game, or a choose-your-own-adventure book, with some paths more likely than others. If we follow a different path, we’ll get different results.
Although the output we see is stochastic, it’s not arbitrary; it’s shaped by the mostly non-stochastic underlying belief distribution. In particular, frontier LLMs do have some introspective ability. There’s a terrific paper from October that shows that experimentally (blog post, paper).
Taking those points into account, I think there’s less difference between LLM and human cognition than it seems. There’s a real need for finding and communicating better mental models for what LLM cognition is actually like, and what implications that has. In my opinion this post is an admirable attempt, and captures the key insight that it’s fairly different than most people imagine, but doesn’t quite get to the right perspective.
I think that this is a really important topic to think about, and we need more people thinking carefully about it, but I have a few points of disagreement:
Consciousness and introspective ability are quite different things. and the presence or absence of introspective ability doesn’t tell us much about whether LLMs are conscious (although, as you point out, the experiment you do here does suggest that their internal states are strange).
It’s important to recall that the output of an LLM isn’t a token; it’s a distribution over token probabilities. The output we see is just the result of following one such path. As a mental model, I like thinking of it as a dialogue tree in a computer game, or a choose-your-own-adventure book, with some paths more likely than others. If we follow a different path, we’ll get different results.
Although the output we see is stochastic, it’s not arbitrary; it’s shaped by the mostly non-stochastic underlying belief distribution. In particular, frontier LLMs do have some introspective ability. There’s a terrific paper from October that shows that experimentally (blog post, paper).
Humans have much less introspective access than we think we do, as demonstrated clearly by experiments on choice blindness. Further, our conscious experience of making a decision (like choosing a number) is something that happens after the decision is made, not during.
Taking those points into account, I think there’s less difference between LLM and human cognition than it seems. There’s a real need for finding and communicating better mental models for what LLM cognition is actually like, and what implications that has. In my opinion this post is an admirable attempt, and captures the key insight that it’s fairly different than most people imagine, but doesn’t quite get to the right perspective.