I’m generally skeptical of evidence of introspective capabilities in LLMs. But even setting that aside, I don’t really understand why ‘introspection’ would ever be theoretically preferable to external white-box methods.
My personal view is that in the limit, introspection could enable far better interpretability than whatever tools we could make, since we can interface directly with the model’s ontology. Of course for transformer LLMs there would be many assumptions to overcome (e.g. is true introspection even possible in the architecture, is it easy to select via gradient descent, will it be hindered by typical LLM priors/scheming) and it doesn’t seem feasible currently
introspection could enable far better interpretability than whatever tools we could make, since we can interface directly with the model’s ontology
I don’t understand why interfacing “directly with the model’s ontology” is best achieved through introspection (even at the limit!). An activation oracle or NL autoencoder also interfaces with the model’s ‘ontology’, and leverages the model’s own features to do so. If anything, my argument above suggests a self-report is not a better channel for the ontology than a tool trained specifically to decode it.
Perhaps we mean different things by ‘ontology’? I take it to be the set of features the model actually represents in (): the directions in activation space it uses to compute. Under that definition, introspection and the oracle read the same object, and introspection is strictly the more lossy approach, since it has to route through the output policy. If instead you mean something the model can access only from the inside, i.e. privileged access, then there’s a double-bind: if that content is in (), the oracle can in principle recover it too; and if it isn’t in (), I don’t see what would make it part of the model’s computation at all. Either way, privileged access fails to distinguish introspection from external decoding.
By ontology I was referring to here and here. I don’t think it has to do with privileged access, rather we might still not have the optimal ways to recover directions or whatever fundamental units that exists in models (assumptions about these fundamentals seem to change, e.g. the platonic representation hypothesis, strong linear-representation hypothesis). AOs and NLAs are essentially based on LLMs interpreting layer activations, but they may be missing considerable detail in this approach, even if they work fairly well. If models can access their activations in a less lossy manner (such that is a significantly less lossy transformation), or internally contain an equivalent AO/NLA, I can imagine introspection functioning better (and better than typical unsupervised methods like SAEs). Of course, this likely doesn’t apply to current LLMs, but this seems like a more scalable approach (by model size or ways that fundamental units are organized) if it holds
You could also provide the entire as an input to this sort of model that knows its ontology very well, and this might be possible in the architecture and easier to compute for—this is arguably not introspection, though this seems inefficient in its own respect
AOs and NLAs are essentially based on LLMs interpreting layer activations, but they may be missing considerable detail in this approach, even if they work fairly well
My intuition is that the opposite premise is true—an AO/NLA should weakly dominate the model’s own introspection; they are fine-tuned clones of the base model’s weights, so they inherit its features and circuitry by construction, including whatever self-modeling the base already does. Then they’re finetuned for the express purpose of reading activations well. So for any introspective signal the base model can surface about its own internal states, a clone optimized for that task can recover the same signal, and the training objective permits it to strictly exceed the base model’s own report. (But perhaps I’m misinterpreting your argument?)
Of course, this likely doesn’t apply to current LLMs, but this seems like a more scalable approach (by model size or ways that fundamental units are organized) if it holds
So there is a funny thing happening here about what is “model” and “not model” (re: current LLMs.) If there is an external symbolic decoder () that reads/takes as input the internal states () of an LLM (), and we consider “model” to be [external symbolic model + LLM] (), then () is self-introspection by definition. But I dislike this, because endlessly expanding the criteria for what counts as a model (e.g., LLM+linear probe, LLM+NLA/AO with a classifier for ‘introspection-related questions’, etc.) seems to strip all functional meaning from the term “introspection.”
My point was moreso that the particular objective used to train NLAs/AOs doesn’t best capture the complete ontology of the base model, and that, in principle, there may be some better objective that only requires the base model reporting from its own introspection
So there is a funny thing happening here about what is “model” and “not model” (re: current LLMs.) If there is an external symbolic decoder () that reads/takes as input the internal states () of an LLM (), and we consider “model” to be [external symbolic model + LLM] (), then () is self-introspection by definition. But I dislike this, because endlessly expanding the criteria for what counts as a model (e.g., LLM+linear probe, LLM+NLA/AO with a classifier for ‘introspection-related questions’, etc.) seems to strip all functional meaning from the term “introspection.”
This is a good point, I agree this kind of setup shouldn’t count as introspection. I was perhaps imagining something like: you can pre-train an LLM such that it involves an explicit self-modelling objective, and it satisfies the objective by implementing circuitry that allows the model to introspect in a highly general manner, native to the model’s ontology, which should be better than trying to guess the best interpretability method ourselves (this is what I meant by whether it is easy to select this sort of introspection by gradient descent. I imagine this is hard to select, for similar reasons as inner search?)
My personal view is that in the limit, introspection could enable far better interpretability than whatever tools we could make, since we can interface directly with the model’s ontology. Of course for transformer LLMs there would be many assumptions to overcome (e.g. is true introspection even possible in the architecture, is it easy to select via gradient descent, will it be hindered by typical LLM priors/scheming) and it doesn’t seem feasible currently
I don’t understand why interfacing “directly with the model’s ontology” is best achieved through introspection (even at the limit!). An activation oracle or NL autoencoder also interfaces with the model’s ‘ontology’, and leverages the model’s own features to do so. If anything, my
Perhaps we mean different things by ‘ontology’? I take it to be the set of features the model actually represents in (
By ontology I was referring to here and here. I don’t think it has to do with privileged access, rather we might still not have the optimal ways to recover directions or whatever fundamental units that exists in models (assumptions about these fundamentals seem to change, e.g. the platonic representation hypothesis, strong linear-representation hypothesis). AOs and NLAs are essentially based on LLMs interpreting layer activations, but they may be missing considerable detail in this approach, even if they work fairly well. If models can access their activations in a less lossy manner (such that
You could also provide the entire
My intuition is that the opposite premise is true—an AO/NLA should weakly dominate the model’s own introspection; they are fine-tuned clones of the base model’s weights, so they inherit its features and circuitry by construction, including whatever self-modeling the base already does. Then they’re finetuned for the express purpose of reading activations well. So for any introspective signal the base model can surface about its own internal states, a clone optimized for that task can recover the same signal, and the training objective permits it to strictly exceed the base model’s own report. (But perhaps I’m misinterpreting your argument?)
So there is a funny thing happening here about what is “model” and “not model” (re: current LLMs.) If there is an external symbolic decoder (
My point was moreso that the particular objective used to train NLAs/AOs doesn’t best capture the complete ontology of the base model, and that, in principle, there may be some better objective that only requires the base model reporting from its own introspection
This is a good point, I agree this kind of setup shouldn’t count as introspection. I was perhaps imagining something like: you can pre-train an LLM such that it involves an explicit self-modelling objective, and it satisfies the objective by implementing circuitry that allows the model to introspect in a highly general manner, native to the model’s ontology, which should be better than trying to guess the best interpretability method ourselves (this is what I meant by whether it is easy to select this sort of introspection by gradient descent. I imagine this is hard to select, for similar reasons as inner search?)