AOs and NLAs are essentially based on LLMs interpreting layer activations, but they may be missing considerable detail in this approach, even if they work fairly well
My intuition is that the opposite premise is true—an AO/NLA should weakly dominate the model’s own introspection; they are fine-tuned clones of the base model’s weights, so they inherit its features and circuitry by construction, including whatever self-modeling the base already does. Then they’re finetuned for the express purpose of reading activations well. So for any introspective signal the base model can surface about its own internal states, a clone optimized for that task can recover the same signal, and the training objective permits it to strictly exceed the base model’s own report. (But perhaps I’m misinterpreting your argument?)
Of course, this likely doesn’t apply to current LLMs, but this seems like a more scalable approach (by model size or ways that fundamental units are organized) if it holds
So there is a funny thing happening here about what is “model” and “not model” (re: current LLMs.) If there is an external symbolic decoder () that reads/takes as input the internal states () of an LLM (), and we consider “model” to be [external symbolic model + LLM] (), then () is self-introspection by definition. But I dislike this, because endlessly expanding the criteria for what counts as a model (e.g., LLM+linear probe, LLM+NLA/AO with a classifier for ‘introspection-related questions’, etc.) seems to strip all functional meaning from the term “introspection.”
My point was moreso that the particular objective used to train NLAs/AOs doesn’t best capture the complete ontology of the base model, and that, in principle, there may be some better objective that only requires the base model reporting from its own introspection
So there is a funny thing happening here about what is “model” and “not model” (re: current LLMs.) If there is an external symbolic decoder () that reads/takes as input the internal states () of an LLM (), and we consider “model” to be [external symbolic model + LLM] (), then () is self-introspection by definition. But I dislike this, because endlessly expanding the criteria for what counts as a model (e.g., LLM+linear probe, LLM+NLA/AO with a classifier for ‘introspection-related questions’, etc.) seems to strip all functional meaning from the term “introspection.”
This is a good point, I agree this kind of setup shouldn’t count as introspection. I was perhaps imagining something like: you can pre-train an LLM such that it involves an explicit self-modelling objective, and it satisfies the objective by implementing circuitry that allows the model to introspect in a highly general manner, native to the model’s ontology, which should be better than trying to guess the best interpretability method ourselves (this is what I meant by whether it is easy to select this sort of introspection by gradient descent. I imagine this is hard to select, for similar reasons as inner search?)
My intuition is that the opposite premise is true—an AO/NLA should weakly dominate the model’s own introspection; they are fine-tuned clones of the base model’s weights, so they inherit its features and circuitry by construction, including whatever self-modeling the base already does. Then they’re finetuned for the express purpose of reading activations well. So for any introspective signal the base model can surface about its own internal states, a clone optimized for that task can recover the same signal, and the training objective permits it to strictly exceed the base model’s own report. (But perhaps I’m misinterpreting your argument?)
So there is a funny thing happening here about what is “model” and “not model” (re: current LLMs.) If there is an external symbolic decoder (
My point was moreso that the particular objective used to train NLAs/AOs doesn’t best capture the complete ontology of the base model, and that, in principle, there may be some better objective that only requires the base model reporting from its own introspection
This is a good point, I agree this kind of setup shouldn’t count as introspection. I was perhaps imagining something like: you can pre-train an LLM such that it involves an explicit self-modelling objective, and it satisfies the objective by implementing circuitry that allows the model to introspect in a highly general manner, native to the model’s ontology, which should be better than trying to guess the best interpretability method ourselves (this is what I meant by whether it is easy to select this sort of introspection by gradient descent. I imagine this is hard to select, for similar reasons as inner search?)