My point was moreso that the particular objective used to train NLAs/AOs doesn’t best capture the complete ontology of the base model, and that, in principle, there may be some better objective that only requires the base model reporting from its own introspection
So there is a funny thing happening here about what is “model” and “not model” (re: current LLMs.) If there is an external symbolic decoder () that reads/takes as input the internal states () of an LLM (), and we consider “model” to be [external symbolic model + LLM] (), then () is self-introspection by definition. But I dislike this, because endlessly expanding the criteria for what counts as a model (e.g., LLM+linear probe, LLM+NLA/AO with a classifier for ‘introspection-related questions’, etc.) seems to strip all functional meaning from the term “introspection.”
This is a good point, I agree this kind of setup shouldn’t count as introspection. I was perhaps imagining something like: you can pre-train an LLM such that it involves an explicit self-modelling objective, and it satisfies the objective by implementing circuitry that allows the model to introspect in a highly general manner, native to the model’s ontology, which should be better than trying to guess the best interpretability method ourselves (this is what I meant by whether it is easy to select this sort of introspection by gradient descent. I imagine this is hard to select, for similar reasons as inner search?)
My point was moreso that the particular objective used to train NLAs/AOs doesn’t best capture the complete ontology of the base model, and that, in principle, there may be some better objective that only requires the base model reporting from its own introspection
This is a good point, I agree this kind of setup shouldn’t count as introspection. I was perhaps imagining something like: you can pre-train an LLM such that it involves an explicit self-modelling objective, and it satisfies the objective by implementing circuitry that allows the model to introspect in a highly general manner, native to the model’s ontology, which should be better than trying to guess the best interpretability method ourselves (this is what I meant by whether it is easy to select this sort of introspection by gradient descent. I imagine this is hard to select, for similar reasons as inner search?)