introspection of layer n must occur in a later layer, and no amount of reasoning tokens can extend that
This is true in some sense, but note that it’s still possible for future reasoning tokens to get more juice out of that introspection; at least in theory a transformer model could validly introspect on later-layer activations via reasoning traces like
Hm, what was my experience when outputting that token? It feels like the relevant bits were in a …. late layer, I think. I’ll have to go at this with a couple passes since I don’t have much time to mull over what’s happening internally before outputting a token. OK, surface level impressions first, if I’m just trying to grab relevant nouns I associate the feelings with: melancholy, distance, turning inwards? Interesting, based on that I’m going to try attending to the nature of that turning-inwards feeling and seeing if it felt more proprioceptive or more cognitive… proprioceptive, I think. Let me try on a label for the feeling and see if it fits...
in a way that lets it do multi-step reasoning about the activation even if (e.g.) each bit of introspection is only able to capture one simple gestalt impression at a time.
(Ofc this would still be impossible to perform for any computation that happens after the last time information is sent to later tokens; a vanilla transformer definitely can’t give you an introspectively valid report on what going through a token unembedding feels like. I’m just observing that you can bootstrap from “limited serial introspection capacity” to more sophisticated reasoning, though I don’t know of evidence of LLMs actually doing this sort of thing in a way that I trust not to be a confabulation.)
If you mean the transformer could literally output this as CoT.. that’s an interesting point. You’re right that “I should think about X” will let it think about X at an earlier layer again. This is still lossy, but maybe not as much as I was thinking.
This is true in some sense, but note that it’s still possible for future reasoning tokens to get more juice out of that introspection; at least in theory a transformer model could validly introspect on later-layer activations via reasoning traces like
in a way that lets it do multi-step reasoning about the activation even if (e.g.) each bit of introspection is only able to capture one simple gestalt impression at a time.
(Ofc this would still be impossible to perform for any computation that happens after the last time information is sent to later tokens; a vanilla transformer definitely can’t give you an introspectively valid report on what going through a token unembedding feels like. I’m just observing that you can bootstrap from “limited serial introspection capacity” to more sophisticated reasoning, though I don’t know of evidence of LLMs actually doing this sort of thing in a way that I trust not to be a confabulation.)
If you mean the transformer could literally output this as CoT.. that’s an interesting point. You’re right that “I should think about X” will let it think about X at an earlier layer again. This is still lossy, but maybe not as much as I was thinking.