Is there a standard term for the following LLM agent behavior?
Agent attempts some task, implements solution A, and writes up a public-facing explanation.
User tells them A is bad, do B instead.
Agent implements B, and writes up a public-facing explanation as if the reader would know they previously tried A.
It’s a quite annoying pattern. I wonder if it’s the result of some kind of overfitting to cases like the following, where a similar-seeming behavior is good: You start out thinking A is a good idea, for the same reasons that many others in your position probably would also think A is a good idea. You try A, realize A doesn’t make sense after all, then in your public-facing writeup about the better idea B, you explain why A in fact isn’t a good idea.
ETA: But the above doesn’t seem to fully explain it. Because in the instances of the bad pattern I have in mind, the agent doesn’t directly explain bad idea A and why it’s bad, rather they write stuff that only makes sense to the reader when A is already salient.
I see this pattern all the time. Claude does this every time I ask it to edit prompts for other agents in some way. In the prompt itself, they will go on and on about what not to do, and why the old version was bad. I constantly have to remind it that the reader doesn’t share the context.
Is there a standard term for the following LLM agent behavior?
Agent attempts some task, implements solution A, and writes up a public-facing explanation.
User tells them A is bad, do B instead.
Agent implements B, and writes up a public-facing explanation as if the reader would know they previously tried A.
It’s a quite annoying pattern. I wonder if it’s the result of some kind of overfitting to cases like the following, where a similar-seeming behavior is good: You start out thinking A is a good idea, for the same reasons that many others in your position probably would also think A is a good idea. You try A, realize A doesn’t make sense after all, then in your public-facing writeup about the better idea B, you explain why A in fact isn’t a good idea.
ETA: But the above doesn’t seem to fully explain it. Because in the instances of the bad pattern I have in mind, the agent doesn’t directly explain bad idea A and why it’s bad, rather they write stuff that only makes sense to the reader when A is already salient.
“Suspiciously specific denial”?
I see this pattern all the time. Claude does this every time I ask it to edit prompts for other agents in some way. In the prompt itself, they will go on and on about what not to do, and why the old version was bad. I constantly have to remind it that the reader doesn’t share the context.