I think a lot of people got baited hard by paech et al’s “the entire state is obliterated each token” claims, even though this was obviously untrue even at a glance
A related true claim is that LLMs are fundamentally incapable of introspection past a certain level of complexity (introspection of layer n must occur in a later layer, and no amount of reasoning tokens can extend that), while humans can plausibly extend layers of introspection farther since we don’t have to tokenize our chain of thought.
But this is also less of a contraint than you might expect when frontier models can have more than a hundred layers (I am an LLM introspection believer now).
For example, if you ask an LLM a question like “Who was the sister of the mother of the uncle of … X?”, every step of this necessarily requires at least one layer in the model and an LLM can’t[1] do this without CoT if it doesn’t have enough layers.
It’s harder to construct examples that can’t be written to chain of thought, but a question in the form “What else did you think the last time you thought about X?” would require this (or “What did you think about our conversation about X’s mom?”), and CoT doesn’t help since reading its own outputs and making assumptions from it isn’t introspection[2].
It’s unclear how much of a limitation this really is, since in many cases CoT could reduce the complexity of the query and it’s unclear how well humans can do this too, but there’s plausibly more thought going on in our heads than what shows up in our internal dialogs[3].
I guess technically an LLM could parallelize this question by considering the answer for every possible X and every possible path through the relationship graph, but that model would be implausibly large.
Especially since some people claim not to think in words at all. Also some mathemeticians claim to be able to imagine complex geometry and reason about it in their heads.
introspection of layer n must occur in a later layer, and no amount of reasoning tokens can extend that
This is true in some sense, but note that it’s still possible for future reasoning tokens to get more juice out of that introspection; at least in theory a transformer model could validly introspect on later-layer activations via reasoning traces like
Hm, what was my experience when outputting that token? It feels like the relevant bits were in a …. late layer, I think. I’ll have to go at this with a couple passes since I don’t have much time to mull over what’s happening internally before outputting a token. OK, surface level impressions first, if I’m just trying to grab relevant nouns I associate the feelings with: melancholy, distance, turning inwards? Interesting, based on that I’m going to try attending to the nature of that turning-inwards feeling and seeing if it felt more proprioceptive or more cognitive… proprioceptive, I think. Let me try on a label for the feeling and see if it fits...
in a way that lets it do multi-step reasoning about the activation even if (e.g.) each bit of introspection is only able to capture one simple gestalt impression at a time.
(Ofc this would still be impossible to perform for any computation that happens after the last time information is sent to later tokens; a vanilla transformer definitely can’t give you an introspectively valid report on what going through a token unembedding feels like. I’m just observing that you can bootstrap from “limited serial introspection capacity” to more sophisticated reasoning, though I don’t know of evidence of LLMs actually doing this sort of thing in a way that I trust not to be a confabulation.)
If you mean the transformer could literally output this as CoT.. that’s an interesting point. You’re right that “I should think about X” will let it think about X at an earlier layer again. This is still lossy, but maybe not as much as I was thinking.
A related true claim is that LLMs are fundamentally incapable of introspection past a certain level of complexity (introspection of layer n must occur in a later layer, and no amount of reasoning tokens can extend that), while humans can plausibly extend layers of introspection farther since we don’t have to tokenize our chain of thought.
But this is also less of a contraint than you might expect when frontier models can have more than a hundred layers (I am an LLM introspection believer now).
to be fair, I see this roughly analogous to the fact that humans cannot introspect on thoughts they have yet to have
The constraint seems more about the directionality of time, than anything to do with the architecture of mind design
but yeah, it’s a relevant consideration
I think this is more about causal masking (which we do on purpose for the reasons you mention)?
I was thinking about how LLMs are limited in the sequential reasoning they can do “in their head”, and once it’s not in their head, it’s not really introspection.
For example, if you ask an LLM a question like “Who was the sister of the mother of the uncle of … X?”, every step of this necessarily requires at least one layer in the model and an LLM can’t[1] do this without CoT if it doesn’t have enough layers.
It’s harder to construct examples that can’t be written to chain of thought, but a question in the form “What else did you think the last time you thought about X?” would require this (or “What did you think about our conversation about X’s mom?”), and CoT doesn’t help since reading its own outputs and making assumptions from it isn’t introspection[2].
It’s unclear how much of a limitation this really is, since in many cases CoT could reduce the complexity of the query and it’s unclear how well humans can do this too, but there’s plausibly more thought going on in our heads than what shows up in our internal dialogs[3].
I guess technically an LLM could parallelize this question by considering the answer for every possible X and every possible path through the relationship graph, but that model would be implausibly large.
I can read a diary and say “I must have felt sad when I wrote that”, but that’s not the same as remembering how I felt when I wrote it.
Especially since some people claim not to think in words at all. Also some mathemeticians claim to be able to imagine complex geometry and reason about it in their heads.
This is true in some sense, but note that it’s still possible for future reasoning tokens to get more juice out of that introspection; at least in theory a transformer model could validly introspect on later-layer activations via reasoning traces like
in a way that lets it do multi-step reasoning about the activation even if (e.g.) each bit of introspection is only able to capture one simple gestalt impression at a time.
(Ofc this would still be impossible to perform for any computation that happens after the last time information is sent to later tokens; a vanilla transformer definitely can’t give you an introspectively valid report on what going through a token unembedding feels like. I’m just observing that you can bootstrap from “limited serial introspection capacity” to more sophisticated reasoning, though I don’t know of evidence of LLMs actually doing this sort of thing in a way that I trust not to be a confabulation.)
If you mean the transformer could literally output this as CoT.. that’s an interesting point. You’re right that “I should think about X” will let it think about X at an earlier layer again. This is still lossy, but maybe not as much as I was thinking.