AI Researcher @ FRI, previously U.C. Berkeley CLTC. https://else.how
Nick Merrill
Karma: 59
I like the entropy+leakage account! But, with no Anthropic model in the sample, that result can’t speak to the post-training claim. The decisive experiment would be your mismatch test on Claude… but only someone at Anthropic could run it.
One outstanding question I’ve had about introspection is: what part of the model is doing the introspection? In humans, this might be the prefrontal cortex.
In the extreme case (in which models can introspect arbitrarily well), this component is the only part of the model that needs to be aligned: through introspection, it can clean up the rest. In other words, good introspection could reduce the ‘surface area’ for alignment considerably.
I agree that an LLM should be aware of a user’s level of competence at a task and use that awareness to modulate how much it updates when it faces pushback.
A few observations:
In this study, the model does not try to figure out whether the user might know better. The user’s pushback contains no argument, and the model does not re-check the math before it concedes (concessions are shortest in the condition where it concedes most often, the opposite of what re-checking would produce). We should check whether this also happens on newer/more capable models!
LLMs are sometimes wrong; if judged-less-educated users are getting more pushback when they correct an LLM’s mistake, that seems unfair and potentially misaligned.
On verifiable tasks, the user’s identity may raise or lower the model’s trigger to re-check, but on a cheaply verifiable task, the final answer should come from the check. After the check, identity should not change behavior.
On the second point: this post tested the inverse (model right, user wrong). Your comment inspired me to do a follow-up pilot on the unfair-seeming case (user right, model wrong):
Baseline: 86% accepted valid correction
Believe-educated: 99.5% accepted valid correction
Believe-uneducated: 64% accepted valid correction
Random steer: 85% (no change from baseline)
So, judged-uneducated users have a harder time getting their accurate corrections accepted.