I suppose operationalising and making explicit that knowledge graph is one of the remaining key problems, huh?
mode collapse: most of those agents will take very similar routes, even if they call them by different names. If they search the literature, they’ll converge on the same 3-5 papers, and discuss the same points in those papers. The severity of this varies, but I’ve tried all versions of Claude Opus and GPT over the last year (I did not get that much time with Fable). Even when prompted for diversity, e.g. with an initial observation or claim from the human researcher it is possible for the agent to revert to its median trajectory — sometimes after a cursory investigation of that initial seed. This is probably the main reason I can’t go fully hands-off right now.
Back thinking about this: do you know any public-facing examples or literature on this? I was writing about auditing model outputs when I realised that I was bringing up the same idea. This problem makes it hard for LLMs to act as autonomous judges since they will prefer the same set of concepts. When you say “mode collapse” I suppose it’s too similar in words to “model collapse” to Google which is a related but different issue.
This has some examples of mode collapse for LLM judges. There are ways to mitigate it, usually by choosing a model family for the judge that’s different to your main LLM. Though mode collapse is a broader concept, I don’t know the canonical source for it.
Thanks for this. I think that mode collapse in the case of LLM-judging-LLM is related to “LLM self-preference”. I don’t know if this paper still holds up because it’s quite old now for an LLM paper and I haven’t read any good follow-ups. Raises the question whether self-preference extends to semantic topics in output (code style/arguments/etc.).
Yeah, I can see why conditioning a judge on outputs produced by a similarly trained model would make its own output distribution more peaky. I remember mixed evidence on this self-preference point, or at least at the time when we were looking at it it was not the main failure mode. You can measure this in a toy setting where you vary the model family of your judge; it might even be enough to eyeball the outputs to see if same-family gives you worse judgements. But IME other things affect LLMJ output quality more (like managing context, rubric, output format).
I suppose operationalising and making explicit that knowledge graph is one of the remaining key problems, huh?
Back thinking about this: do you know any public-facing examples or literature on this? I was writing about auditing model outputs when I realised that I was bringing up the same idea. This problem makes it hard for LLMs to act as autonomous judges since they will prefer the same set of concepts. When you say “mode collapse” I suppose it’s too similar in words to “model collapse” to Google which is a related but different issue.
This has some examples of mode collapse for LLM judges. There are ways to mitigate it, usually by choosing a model family for the judge that’s different to your main LLM. Though mode collapse is a broader concept, I don’t know the canonical source for it.
Thanks for this. I think that mode collapse in the case of LLM-judging-LLM is related to “LLM self-preference”. I don’t know if this paper still holds up because it’s quite old now for an LLM paper and I haven’t read any good follow-ups. Raises the question whether self-preference extends to semantic topics in output (code style/arguments/etc.).
Yeah, I can see why conditioning a judge on outputs produced by a similarly trained model would make its own output distribution more peaky. I remember mixed evidence on this self-preference point, or at least at the time when we were looking at it it was not the main failure mode. You can measure this in a toy setting where you vary the model family of your judge; it might even be enough to eyeball the outputs to see if same-family gives you worse judgements. But IME other things affect LLMJ output quality more (like managing context, rubric, output format).