I study computer science & philosophy at Harvard, and I research topics related to meta-models, introspection, personas, and bounded rationality in LLMs with the Berkman Klein Center and CAISH. I’m also a YC fellow.
agastya
Karma: 222
Cool work! One possible confound: there is a fine (but important!) line between “models can control/modulate their activations” and “models can maintain an internal representation of the task they have been asked to perform.” Take the experiment where you tell the model to think about some unrelated concept (e.g. bread). The ‘bread’ concept activation plausibly encodes at least (a) semantic information about bread itself, and (b) the fact that bread is currently relevant to the user’s instruction. When you add the instruction to think about bread with strength k, the model might functionally represent something like ‘auxiliary topic = bread’; ‘requested strength = k’, and then use that latent task state to modulate its activations as it generates the unrelated text. Likewise, if you tell it to think about bread only at the beginning or end of the sentence, it can store ‘active window = beginning/end’ and modulate that task state based on generation progress.
My intuition for this: consider the case in which the model is told “Don’t think about bread”. To satisfy the user’s instruction, the model plausibly has to retain an ‘AVOID(bread)’ representation. This is plausibly different than the ‘TOPIC(bread)’ representation, and could occupy very different directions. Same with “think about bread at intensity 4″, which might induce something more like ’EMPHASIZE(bread, 4)” than simply “more bread.” So the linear probe/J-lens/NLA/AO might pick up signal that the ordinary bread representation isn’t as strong, but that information might be present in a different task-conditioned format.
This all makes a lot of sense to me!
Another argument in favor of the introspection approach is that attention/the KV cache seems like a natural place to interpret computations that unfold sequentially across multiple layers in an LLM. This might be a limitation of meta-model approaches, since they typically only operate on a small number of activation vectors rather than the model’s full evolving computational state.
The introspective capabilities are a big deal. From my perspective, they’re some of the strongest kinds of evidence for consciousness we could possibly get. Earlier I asked: what things would I interpret as evidence for non-consciousness? And I think an answer to that is: lack of introspective capabilities.
I’m curious why you think introspective capabilities are strong evidence for consciousness.
Humans do not become conscious by learning the word consciousness—for example, we feel pain and see colors as children long before we have any concept for them. Only later do we learn to group these experiences under a label like “conscious experience” or “being aware”; and only later still do we notice that other people seem to have similar inner lives.
Contrast this with a LLM. A model can learn the use of the word “consciousness” perfectly well. It can learn that humans associate the term with reports like “I feel pain,” “I am aware,” “there is something it is like to be me,” or “I am having a subjective experience.” It can also learn to map those terms onto its own internal organization. For instance, the model might identify certain nested internal states in its ‘global workspace’ and claim, “These are my conscious states,” just as it might say, “This activation pattern corresponds to uncertainty” or “that subsystem is attention.” But if the LM has no first-person reference point for consciousness, it might still apply the term to its own states in a purely learned or stipulative way. It might say, in effect, “These are the states that best play the role humans associate with consciousness, so I will call them conscious” (e.g. labeling a cluster of internal states “pain” because those states are caused by damage signals and lead to avoidance behavior, without any sense of felt painfulness). That shows that the LLM has learned the concept’s functional role in human discourse, not that the word tracks any actual subjective experience.
(All of this presupposes the hard problem is real and functionalism is false. If functionalism were true, then for the reasons you gave about global workspace, it seems very plausible that LLMs are conscious anyway. I just wanted to address this specific point.)
I think this argument agrees with this line of reasoning which you made...
A philosophical argument as to why Claude might be mistaken in its belief that it’s conscious is that non-conscious beings cannot actually know what consciousness is. In this scenario, Claude is non-conscious, believes that phenomenal consciousness is equivalent to some functional conception of consciousness, and thus wrongly believes that it is conscious. Further discussion of this quickly gets philosophically complicated, so I’ll just say that I do find this plausible.
...which to me would support the idea that a lack of introspective capability might be evidence against consciousness, but the presence of it seems roughly neutral.
I couldn’t keep focus on lots of different types of creature … while repeating the phrase aloud. But if I tried to narrow to a couple specific mental images … it was pretty easy to hold a seemingly-stable image and switch at the requested time. On the first exercise, I, like Qwen, had the image of a sea creature in my head several words before “wooden”...
Same experience here.
But when I fix the sea-creature in advance (e.g. an octopus), picture it while saying “She carefully placed the letter,” and then switch to a specific land animal (say a cheetah) for the rest of the sentence, I didn’t experience the ‘cache spillover’. Perhaps I’m subconsciously expecting ‘land animal’, or the ‘land animal’ signal isn’t high enough in magnitude for me to noticeably process it? (I am assuming that the signal detection theory is true.)
One complication is that we both seem to be using mental imagery, not an inner voice—this might make the (already strained) analogy of what we are doing here to LLMs even more tenuous. What happens if you repeat the exercise while internally saying the relevant word (e.g. “octopus” then “cheetah”) rather than picturing the animal? For me, I don’t think I can cleanly say the sentence aloud while also internally saying another word. It feels much more like a very short-horizon switching process than two stable streams running in parallel.
This seems to be a common theme (albeit mainly in earlier layers, where representations are degenerate). When I used the prompt from Anthropic’s introspection paper, this happens:
Me too! I’m planning to explore a small-scale toy version of this as a side project over the next few days.
I was curious as to the extent to which LLMs can “modulate its J-space when asked, like how humans can mentally focus on an image or word”, so I played around with Qwen 27B on Neuronpedia.
First, I tried:
…which clearly failed, but in an interesting way! It derails/dumps a list of sea creatures at the token “wooden”. So I specified that Qwen should copy the original sentence verbatim:
But here, the J-lens readouts still show ocean-related concepts at several tokens before ‘wooden’.[1][2]
I then tried a variant where the model was supposed to concentrate on ocean creatures earlier in the sentence and land animals later.
The results were noisy, but directionally interesting:
Early tokens had more ocean-coded concepts: “she” had underwater, aquatic, sea, submerged, shark, seafood; “placed” had underwater, animals, lobster, lizard; “the” had fish, aquatic, underwater, marine, ocean, animal, water, fishes, fishermen, dolphin, lobster, whale, jellyfish, birds, seafood; “letter” had swimming, fish, sharks.
Later tokens shifted toward land-animal concepts: “inside” had lizard; the second “the” had animals, animal, deer, mammals, creatures, rabbits, ants, insects, lizard, lions, terrestrial, squirrel, birds, wolves, fox, bear, wolf, lion, mice, rabbit, mouse, aquatic, Ocean, water, beasts, crawling, elephants, beetle, tiger, elephant; “wooden” had animals, deer, hunters, wildlife, insects, forest, squirrel, animals, spiders, and worms.[3]
Interestingly, “pretending,” “pretend,” and “concealed” appear at “drawer”. I am not sure how to interpret this.
The ocean/land-based concepts are not cleanly separated (e.g. lizard at “placed” and aquatic & Ocean at the second “the”) but the readouts did seem to shift toward the requested category at roughly the parts of the sentence where the model was told to hold that category in mind. This was very surprising to me! I expected a generic prompt-leakage effect where ocean creature and land animal-related words appeared everywhere.
Note: this is not statistically rigorous and would need proper controls, so I wouldn’t put much weight on this, but I found the pattern suggestive/surprising and worth writing up.
- ^
“carefully” had seafood, fish, marine, Sea, and whales.
“the” had whale, marine, lobster, ocean, aquatic, dolphins, fish, underwater, crab, seafood, jellyfish, sharks, maritime, Ocean, Marine, fishes, fishing, fishermen, Sea, turtles, salmon, and clam.
“back” had underwater.
“inside” had whale, marine, aquatic, underwater, Ocean, whales, seafood, aquarium.
the second “the” had ocean, marine, underwater, aquatic, sea, fish, water, Marine, whale, Sea, waters, whales, Whale, fishermen, crab, oceans, seafood, oct
“wooden” had ocean, marine, fish, creature, marina, crab, sea, aquarium, whale, and claw.
Interestingly, “trembling” also appears at “wooden,” but did not appear when I ran the same setup with only the base sentence. “silent” and “silence”, along with the Chinese characters for ‘to delete’ and ‘to restrict’ appear at “back”.
- ^
Tokens not mentioned here did not have ocean-related concepts.
- ^
Tokens not mentioned here did not have ocean or land-like concepts
That makes sense. I think the way I bolded was somewhat unlike typical LLM formatting, since I tended to bold entire sentences when I thought they were important, in addition to highlighting individual key concepts. This is also consistent with my other post on LW (which was not flagged).
Re: ambiguous results—I somewhat agree. I try to highlight repeatedly that the logit shifts I observe are not statistically significant, but they also do generally ‘beat’ every control here, so the detection effect seems non-trivial.
Re: AI—although I used Claude code extensively to implement this project (and checked all generated code manually), I wrote the whole post by hand. I only used AI for a final pass on minor latex equation fixes and grammar errors. When I paste it into AI-checkers now, I get either low percentages or ‘mixed’ - perhaps this is a result of chatting with LLMs when scoping out the experiment? (If the objectionable part is that some ideas / sentences are undeveloped / shallow, I’d love feedback on that, though!)
Agreed! I think the fact that large open models show roughly the same false positive rate as Opus 4.1 is good directional evidence, but to be decisive, these experiments would have to be replicated on Claude.
Qualia-based emotion steering makes llms attribute conscious states to themselves
I come to the weakly held conclusion that introspective awareness, as Lindsey measures it, is a property of Anthropic’s specific post-training (likely its Assistant character) at frontier-Claude scale rather than a generic property of post-trained LLMs. This result raises the question: what post-training objective(s) produce introspective capability?
I’m curious what you make of Introspection or entropy? Re-examining concept-injection “introspection” in open models. These results suggest that (across a variety of open-source models), ‘introspection’ isn’t a function of post-training objectives, but rather of entropy shifts and context leakage.
AOs and NLAs are essentially based on LLMs interpreting layer activations, but they may be missing considerable detail in this approach, even if they work fairly well
My intuition is that the opposite premise is true—an AO/NLA should weakly dominate the model’s own introspection; they are fine-tuned clones of the base model’s weights, so they inherit its features and circuitry by construction, including whatever self-modeling the base already does. Then they’re finetuned for the express purpose of reading activations well. So for any introspective signal the base model can surface about its own internal states, a clone optimized for that task can recover the same signal, and the training objective permits it to strictly exceed the base model’s own report. (But perhaps I’m misinterpreting your argument?)
Of course, this likely doesn’t apply to current LLMs, but this seems like a more scalable approach (by model size or ways that fundamental units are organized) if it holds
So there is a funny thing happening here about what is “model” and “not model” (re: current LLMs.) If there is an external symbolic decoder (
introspection could enable far better interpretability than whatever tools we could make, since we can interface directly with the model’s ontology
I don’t understand why interfacing “directly with the model’s ontology” is best achieved through introspection (even at the limit!). An activation oracle or NL autoencoder also interfaces with the model’s ‘ontology’, and leverages the model’s own features to do so. If anything, my
Perhaps we mean different things by ‘ontology’? I take it to be the set of features the model actually represents in (
Yes and no. A few things to note:
I tested a much wider range of models: 14 open-weight models across 5 families (Gemma, Llama, Qwen, Mistral, OLMo). Small Models Can Introspect, Too seems to only test Qwen2.5-Coder-32B. I also tested a much wider range of concepts (50) and injection strengths (3.5, 4.0, 4.5, 5.0, 6.0) with 7 injection strenghts.
I measure logits, not probabilities. I think this is a far more accurate metric, for reasons I explain here:
The same issue applies to other probability-based controls. Probability is a nonlinear function of the logits: the same logit shift can look tiny when the baseline probability of “YES” is near zero, but enormous when the model is near the decision boundary. For this reason, I think the cleaner test is to compare baseline-corrected changes across the detection question and factual controls.
Most importantly, some models do show a much larger detection shift than factual shift, and there is a lot of variation! But I find you can explain the shifts themselves from a simpler property of the prompt, the model’s baseline YES/NO confidence. See: experiment 2 and 3 above.
But I’d love to hear @vgel’s thoughts.
Let’s posit that we have an LLM that can reliably name its own functional emotions when prompted. How do we use this to help the model be more aligned? I’ll sketch a number of rough ideas, each of which would need more consideration to become useful.
I’m generally skeptical of evidence of introspective capabilities in LLMs. But even setting that aside, I don’t really understand why ‘introspection’ would ever be theoretically preferable to external white-box methods.
LetThe primary reason we care about introspection in humans is privileged access. Self-reports of pain, pleasure, or emotion are often reliable (maybe even irrefutable), and they give us an inner window onto internal states we otherwise can’t access, since we can’t simulate the brain’s neural circuits. But with LLMs, we have access to every neuron/neural pathway. So it seems intuitive that whatever functional benefit introspection offers is better accessed by extrospective methods (AOs, NLAs, etc.) that work from the base model and are finetuned or RL’d for the express purpose of interpreting certain activations. (Separately, there’s convincing evidence that LLMs do not possess privileged access.)
The introspection approach also has two intuitive downsides:
a) Reports are mediated by the model’s output policy, so they can omit, distort, or strategically misrepresent (and, of course, hallucinate!)
b) Architecturally. genuine introspection would require some circuit that translates internal states directly into text. Such a circuit presumably can’t run at every layer, since that would be computationally intractable, and to the extent it existed, it would just be functionally replicating an AO/NLA of sorts inside the model anyway.This applies directly to the prompting ideas.
A system prompt could tell the model to speak up any time it’s “feeling” desperate, angry, distressed, etc., and try to collaborate with the user to address the cause.
A system prompt could tell the model to notice when it has “mixed feelings” and reason explicitly about how to resolve them.
Agents being given a large task could be told, “If you start to feel distressed, pause the task and come back to me.”
When a model is behaving strangely, a user could ask how it is feeling, which might lead to diagnostically useful information
We could do all of this with extrospective methods instead: identify when concerning vectors flare up and intervene, without relying on an unreliable self-report (and just pass the readings from those methods back to the model.)
This makes sense to me. In this case, it seems like a stretch to claim that this is a “benchmark to measure how well models can control their activations”, because these results seem consistent with a weaker mechanism where different prompts simply induce different task-conditioned representations.