Thanks for your thoughts. I’m thinking along the same lines for biological research, where the ground-truth is very far away and many analytical outcomes can seem plausible.
I am deliberately assuming humans as the final decision-makers, and asking how far we can get by augmenting human judgement
I agree with this stance and I think we should keep humans in the loop as much as possible. But I wonder what the response should be if you run several AI agents on the same topic that come to different conclusions. (Although it seems like you have the opposite problem of them all doing exactly the same thing).
are differences in productivity between academic fields
I think even looking within subfields (say, within biology), the rate of uptake of similar ideas can be very different. Some fields are just more amenable to disproving ideas more than others, considering the availability of evidence and the possible variation created by experiment. I would expect AI to make the gaps in theory smaller because synthesis across fields is easier, but I don’t think it will change the fundamental differences in experimental design.
But I wonder what the response should be if you run several AI agents on the same topic that come to different conclusions.
I would guess that situations where answering one question raises a few others are fairly common. You can have conflicting evidence and that’s a pointer that your understanding is somehow off, so you have to dig deeper. If you keep following those edges on the knowledge graph and still have compelling evidence going both ways, something might be wrong with your world model such that you need to reevaluate more fundamental assumptions. It shouldn’t be possible to arrive at different conclusions if your work so far is correct. This is one way in which massively parallelising the legwork doesn’t speed you up – it’s a sort of Amdahl’s law effect – because you still need to understand the assumptions, context, etc.
To that extent you want to reduce the verification cost of each agent thread. That’s why I briefly touched on debate as a method of eliciting the crux. My sense is that if the crux is obvious to the human reviewer progress is more easily made. If it isn’t, the human reviewer has to do increasingly more costly tasks to review the work, potentially up to the cost of generating it in the first place.
I suppose operationalising and making explicit that knowledge graph is one of the remaining key problems, huh?
mode collapse: most of those agents will take very similar routes, even if they call them by different names. If they search the literature, they’ll converge on the same 3-5 papers, and discuss the same points in those papers. The severity of this varies, but I’ve tried all versions of Claude Opus and GPT over the last year (I did not get that much time with Fable). Even when prompted for diversity, e.g. with an initial observation or claim from the human researcher it is possible for the agent to revert to its median trajectory — sometimes after a cursory investigation of that initial seed. This is probably the main reason I can’t go fully hands-off right now.
Back thinking about this: do you know any public-facing examples or literature on this? I was writing about auditing model outputs when I realised that I was bringing up the same idea. This problem makes it hard for LLMs to act as autonomous judges since they will prefer the same set of concepts. When you say “mode collapse” I suppose it’s too similar in words to “model collapse” to Google which is a related but different issue.
This has some examples of mode collapse for LLM judges. There are ways to mitigate it, usually by choosing a model family for the judge that’s different to your main LLM. Though mode collapse is a broader concept, I don’t know the canonical source for it.
Thanks for this. I think that mode collapse in the case of LLM-judging-LLM is related to “LLM self-preference”. I don’t know if this paper still holds up because it’s quite old now for an LLM paper and I haven’t read any good follow-ups. Raises the question whether self-preference extends to semantic topics in output (code style/arguments/etc.).
Yeah, I can see why conditioning a judge on outputs produced by a similarly trained model would make its own output distribution more peaky. I remember mixed evidence on this self-preference point, or at least at the time when we were looking at it it was not the main failure mode. You can measure this in a toy setting where you vary the model family of your judge; it might even be enough to eyeball the outputs to see if same-family gives you worse judgements. But IME other things affect LLMJ output quality more (like managing context, rubric, output format).
Thanks for your thoughts. I’m thinking along the same lines for biological research, where the ground-truth is very far away and many analytical outcomes can seem plausible.
I agree with this stance and I think we should keep humans in the loop as much as possible. But I wonder what the response should be if you run several AI agents on the same topic that come to different conclusions. (Although it seems like you have the opposite problem of them all doing exactly the same thing).
I think even looking within subfields (say, within biology), the rate of uptake of similar ideas can be very different. Some fields are just more amenable to disproving ideas more than others, considering the availability of evidence and the possible variation created by experiment. I would expect AI to make the gaps in theory smaller because synthesis across fields is easier, but I don’t think it will change the fundamental differences in experimental design.
I would guess that situations where answering one question raises a few others are fairly common. You can have conflicting evidence and that’s a pointer that your understanding is somehow off, so you have to dig deeper. If you keep following those edges on the knowledge graph and still have compelling evidence going both ways, something might be wrong with your world model such that you need to reevaluate more fundamental assumptions. It shouldn’t be possible to arrive at different conclusions if your work so far is correct. This is one way in which massively parallelising the legwork doesn’t speed you up – it’s a sort of Amdahl’s law effect – because you still need to understand the assumptions, context, etc.
To that extent you want to reduce the verification cost of each agent thread. That’s why I briefly touched on debate as a method of eliciting the crux. My sense is that if the crux is obvious to the human reviewer progress is more easily made. If it isn’t, the human reviewer has to do increasingly more costly tasks to review the work, potentially up to the cost of generating it in the first place.
I suppose operationalising and making explicit that knowledge graph is one of the remaining key problems, huh?
Back thinking about this: do you know any public-facing examples or literature on this? I was writing about auditing model outputs when I realised that I was bringing up the same idea. This problem makes it hard for LLMs to act as autonomous judges since they will prefer the same set of concepts. When you say “mode collapse” I suppose it’s too similar in words to “model collapse” to Google which is a related but different issue.
This has some examples of mode collapse for LLM judges. There are ways to mitigate it, usually by choosing a model family for the judge that’s different to your main LLM. Though mode collapse is a broader concept, I don’t know the canonical source for it.
Thanks for this. I think that mode collapse in the case of LLM-judging-LLM is related to “LLM self-preference”. I don’t know if this paper still holds up because it’s quite old now for an LLM paper and I haven’t read any good follow-ups. Raises the question whether self-preference extends to semantic topics in output (code style/arguments/etc.).
Yeah, I can see why conditioning a judge on outputs produced by a similarly trained model would make its own output distribution more peaky. I remember mixed evidence on this self-preference point, or at least at the time when we were looking at it it was not the main failure mode. You can measure this in a toy setting where you vary the model family of your judge; it might even be enough to eyeball the outputs to see if same-family gives you worse judgements. But IME other things affect LLMJ output quality more (like managing context, rubric, output format).