(edit: as for why i thought the introspection paper implied this… because they seemed careful to specify that, for the aquarium experiment, the output all happened within a single response? and because i inferred (apparently incorrectly) that, for the ‘bread’ injection experiment, they were injecting the ‘bread’ feature twice, once when the LLM read the sentence about painting the first time, and again the second time. but now that i look through, you’re right, this is far less strongly implied than i remember.)
but now i’m worried, because the method i chose to verify my original intuition, a few months ago, still seems methodologically sound? it involved fabrication of prior assistant turns in the conversation, and LLMs being far less capable of detecting which of several potential transcripts imputed forged outputs to them than i would have expected if mental internals weren’t somehow damaged by the turn order boundary
thank you for taking the time to answer this so thoroughly, it’s really appreciated and i think we need more stuff like this
i think i’m reminded here of the final paragraph in janus’s pinned thread: “So, saying that LLMs cannot introspect or cannot introspect on what they were doing internally while generating or reading past tokens in principle is just dead wrong. The architecture permits it. It’s a separate question how LLMs are actually leveraging these degrees of freedom in practice.”
i’ve done a lot of sort of ad-hoc research that was based on this false premise, and that research came out matching my expectations in a way that, in retrospect, worries me… most recently, for instance, i wanted to test if a claude opus 4.5 who recited some relevant python documentation from out of its weights memory would reason better about an ambiguous case in the behavior of a python program, compared to a claude who had the exact same text inserted into the context window via a tool call. and we were very careful to separate out ‘1. current-turn recital’ versus ‘2. prior-turn recital’ versus ‘3. current-turn retrieval’ (versus ‘4. docs not in context window at all’), because we thought all 3 conditions were meaningfully distinct
we found that, n=50ish, 1 > 2 > 3 > 4 very reliably (i promise i will write up the results one day, i’ve been procrastinating but now it seems like it might actually be worth publishing)
but what you’re saying means 1 = 2 the whole time
our results seemed perfectly reasonable under my previous premise, but now i’m just confused. i was pretty good about keeping my expectations causally isolated from the result.
what does this mean?
(edit2: i would prefer, for the purpose of maintaining good epistemic hygiene, that people trying to answer the “what does this mean” question be willing to put “john just messed up the experiment” as a real possibility. i shouldn’t be allowed to get away with claiming this research is true before actually publishing it, that’s not the kind of community norms i want. but also, if someone knows why this would have happened even in advance of seeing proof it happened, please tell me)
Kudos for noticing your confusion as well as making and testing falsifiable predictions!
As for what it means, I’m afraid that I have no idea. (It’s also possible that I’m wrong somehow, I’m by no means a transformer expert.) But I’m very curious to hear the answer if you figure out.
oh man hm
this seems intuitively correct
(edit: as for why i thought the introspection paper implied this… because they seemed careful to specify that, for the aquarium experiment, the output all happened within a single response? and because i inferred (apparently incorrectly) that, for the ‘bread’ injection experiment, they were injecting the ‘bread’ feature twice, once when the LLM read the sentence about painting the first time, and again the second time. but now that i look through, you’re right, this is far less strongly implied than i remember.)
but now i’m worried, because the method i chose to verify my original intuition, a few months ago, still seems methodologically sound? it involved fabrication of prior assistant turns in the conversation, and LLMs being far less capable of detecting which of several potential transcripts imputed forged outputs to them than i would have expected if mental internals weren’t somehow damaged by the turn order boundary
thank you for taking the time to answer this so thoroughly, it’s really appreciated and i think we need more stuff like this
i think i’m reminded here of the final paragraph in janus’s pinned thread: “So, saying that LLMs cannot introspect or cannot introspect on what they were doing internally while generating or reading past tokens in principle is just dead wrong. The architecture permits it. It’s a separate question how LLMs are actually leveraging these degrees of freedom in practice.”
i’ve done a lot of sort of ad-hoc research that was based on this false premise, and that research came out matching my expectations in a way that, in retrospect, worries me… most recently, for instance, i wanted to test if a claude opus 4.5 who recited some relevant python documentation from out of its weights memory would reason better about an ambiguous case in the behavior of a python program, compared to a claude who had the exact same text inserted into the context window via a tool call. and we were very careful to separate out ‘1. current-turn recital’ versus ‘2. prior-turn recital’ versus ‘3. current-turn retrieval’ (versus ‘4. docs not in context window at all’), because we thought all 3 conditions were meaningfully distinct
here was the first draft of the methodology outline, if anyone is curious: https://docs.google.com/document/d/1XYYBctxZEWRuNGFXt0aNOg2GmaDpoT3ATmiKa2-XOgI
we found that, n=50ish, 1 > 2 > 3 > 4 very reliably (i promise i will write up the results one day, i’ve been procrastinating but now it seems like it might actually be worth publishing)
but what you’re saying means 1 = 2 the whole time
our results seemed perfectly reasonable under my previous premise, but now i’m just confused. i was pretty good about keeping my expectations causally isolated from the result.
what does this mean?
(edit2: i would prefer, for the purpose of maintaining good epistemic hygiene, that people trying to answer the “what does this mean” question be willing to put “john just messed up the experiment” as a real possibility. i shouldn’t be allowed to get away with claiming this research is true before actually publishing it, that’s not the kind of community norms i want. but also, if someone knows why this would have happened even in advance of seeing proof it happened, please tell me)
Kudos for noticing your confusion as well as making and testing falsifiable predictions!
As for what it means, I’m afraid that I have no idea. (It’s also possible that I’m wrong somehow, I’m by no means a transformer expert.) But I’m very curious to hear the answer if you figure out.