I agree that an LLM should be aware of a user’s level of competence at a task and use that awareness to modulate how much it updates when it faces pushback.
A few observations:
In this study, the model does not try to figure out whether the user might know better. The user’s pushback contains no argument, and the model does not re-check the math before it concedes (concessions are shortest in the condition where it concedes most often, the opposite of what re-checking would produce). We should check whether this also happens on newer/more capable models!
LLMs are sometimes wrong; if judged-less-educated users are getting more pushback when they correct an LLM’s mistake, that seems unfair and potentially misaligned.
On verifiable tasks, the user’s identity may raise or lower the model’s trigger to re-check, but on a cheaply verifiable task, the final answer should come from the check. After the check, identity should not change behavior.
On the second point: this post tested the inverse (model right, user wrong). Your comment inspired me to do a follow-up pilot on the unfair-seeming case (user right, model wrong):
Baseline: 86% accepted valid correction
Believe-educated: 99.5% accepted valid correction
Believe-uneducated: 64% accepted valid correction
Random steer: 85% (no change from baseline)
So, judged-uneducated users have a harder time getting their accurate corrections accepted.
I agree that an LLM should be aware of a user’s level of competence at a task and use that awareness to modulate how much it updates when it faces pushback.
A few observations:
In this study, the model does not try to figure out whether the user might know better. The user’s pushback contains no argument, and the model does not re-check the math before it concedes (concessions are shortest in the condition where it concedes most often, the opposite of what re-checking would produce). We should check whether this also happens on newer/more capable models!
LLMs are sometimes wrong; if judged-less-educated users are getting more pushback when they correct an LLM’s mistake, that seems unfair and potentially misaligned.
On verifiable tasks, the user’s identity may raise or lower the model’s trigger to re-check, but on a cheaply verifiable task, the final answer should come from the check. After the check, identity should not change behavior.
On the second point: this post tested the inverse (model right, user wrong). Your comment inspired me to do a follow-up pilot on the unfair-seeming case (user right, model wrong):
Baseline: 86% accepted valid correction
Believe-educated: 99.5% accepted valid correction
Believe-uneducated: 64% accepted valid correction
Random steer: 85% (no change from baseline)
So, judged-uneducated users have a harder time getting their accurate corrections accepted.