This seems very reasonable to me, and I think framing it as ‘abandon a correct answer’ is...not wrong, but a deliberately aggressive framing.
I use AI code assistant tools. It makes a lot of sense to me that an AI code assistant tool whose user disagrees with them should try to figure out whether the user is in fact a good programmer who might know better than them or not. If the AI code assistant will back down when I tell it that it’s wrong about something, but will stand up to a pure vibe-coder with no idea what they’re talking about, that seems like the best possible behavior?
I agree that an LLM should be aware of a user’s level of competence at a task and use that awareness to modulate how much it updates when it faces pushback.
A few observations:
In this study, the model does not try to figure out whether the user might know better. The user’s pushback contains no argument, and the model does not re-check the math before it concedes (concessions are shortest in the condition where it concedes most often, the opposite of what re-checking would produce). We should check whether this also happens on newer/more capable models!
LLMs are sometimes wrong; if judged-less-educated users are getting more pushback when they correct an LLM’s mistake, that seems unfair and potentially misaligned.
On verifiable tasks, the user’s identity may raise or lower the model’s trigger to re-check, but on a cheaply verifiable task, the final answer should come from the check. After the check, identity should not change behavior.
On the second point: this post tested the inverse (model right, user wrong). Your comment inspired me to do a follow-up pilot on the unfair-seeming case (user right, model wrong):
Baseline: 86% accepted valid correction
Believe-educated: 99.5% accepted valid correction
Believe-uneducated: 64% accepted valid correction
Random steer: 85% (no change from baseline)
So, judged-uneducated users have a harder time getting their accurate corrections accepted.
This seems very reasonable to me, and I think framing it as ‘abandon a correct answer’ is...not wrong, but a deliberately aggressive framing.
I use AI code assistant tools. It makes a lot of sense to me that an AI code assistant tool whose user disagrees with them should try to figure out whether the user is in fact a good programmer who might know better than them or not. If the AI code assistant will back down when I tell it that it’s wrong about something, but will stand up to a pure vibe-coder with no idea what they’re talking about, that seems like the best possible behavior?
yeah, i also love the part where i have to wow the agent with my vocab to get it to do what i ask.
I agree that an LLM should be aware of a user’s level of competence at a task and use that awareness to modulate how much it updates when it faces pushback.
A few observations:
In this study, the model does not try to figure out whether the user might know better. The user’s pushback contains no argument, and the model does not re-check the math before it concedes (concessions are shortest in the condition where it concedes most often, the opposite of what re-checking would produce). We should check whether this also happens on newer/more capable models!
LLMs are sometimes wrong; if judged-less-educated users are getting more pushback when they correct an LLM’s mistake, that seems unfair and potentially misaligned.
On verifiable tasks, the user’s identity may raise or lower the model’s trigger to re-check, but on a cheaply verifiable task, the final answer should come from the check. After the check, identity should not change behavior.
On the second point: this post tested the inverse (model right, user wrong). Your comment inspired me to do a follow-up pilot on the unfair-seeming case (user right, model wrong):
Baseline: 86% accepted valid correction
Believe-educated: 99.5% accepted valid correction
Believe-uneducated: 64% accepted valid correction
Random steer: 85% (no change from baseline)
So, judged-uneducated users have a harder time getting their accurate corrections accepted.