I think this paper over-emphasizes the risk of miscalibration and underemphasizes the risk of bad evidence.
When you delegate work to another entity (AI, human, orgs), you usually do not just replace your work by theirs and then propagate updates in the same way you would have done if you trusted them as much as you. You usually update less hard because the fact that they are not as trusted as you is priced-in. And so the sort of argument tree with mistakes in them causing downstream mistakes that this paper presents is not how I expect it to look like in practice.
In practice (in the sort of Slopolis world that this paper focuses on), I expect humans to notice some slop, know that slop exists, but not be able to know exactly where it is, and thus update much less hard on AI-generated results than on results in which they were heavily involved, similar to
how you would update less hard on the good results of a fancy technique in an academic paper because you know that the actual results are often less good than they seem
how you would update less on your own results if you did them sloppily
how people already don’t trust AI-generated code that much and ask for stronger evidence about code correctness than if a human wrote it
While this can result in some miscalibration due to confirmation bias (which can result in people updating on bad evidence), I think the main effect this will have is a large degradation in effective productivity. I expect it to look something like “you have millions of AI-generated paper-size contributions that look as good as human-generated ones, but you don’t trust them that much because they are AI-generated, and so the size of your update away from your priors is only as good as if you had thousands of paper-size human contributions”.
This also means that I expect people to stay closer to their priors than what you might have hoped based on the volume of research that will happen. And so if you have different priors from decision makers, it might feel like they are miscalibrated, when in fact most of the effect size may just be that they had different priors from you and they (correctly) didn’t update that much given that the evidence was bad.
Thanks for the comment! I think bad evidence, or the difficulty of knowing how much to trust AI-generated research is probably a core issue with automating the work.
I wonder how much room there is to increase the signal:noise ratio of AI-generated work by building a good interface (such that even if you are still updating less than on the equivalent volume of work from humans, you still get good value-for-money from running lots of parallel agents). I’m assuming a world where lots of (empirical) alignment work can be done by agents; in that world, human researchers are the bottleneck. How do you help those researchers make sense of the work done by agents?
I’ve been prototyping this, but so far I haven’t found a version that I personally find useful. It seems to devolve into imposing a structure: a research “contract” that specifies the hypothesis and work to be delivered, artefacts which have a 1-click replication script, an interface that displays all this together with any shared assumptions (to potentially discount correlated evidence) or resources (e.g. if 3 workstreams use the same dataset, if there is something off about that dataset all 3 are invalidated), and so on. I’m imagining easy replication via other agents is also useful, as well as any other work that would lift the ceiling of what a person can reasonably oversee (pairing the main worker with a “falsifier” agent, LLM as a judge hooked in at various points...).
I think some of these could work. That said, I have a high bar for replacing my default agent-assisted workflow, and this hasn’t cleared it yet. I’m curious if you think there is (and if so, where) value in improving the interaction surface.
I think this paper over-emphasizes the risk of miscalibration and underemphasizes the risk of bad evidence.
When you delegate work to another entity (AI, human, orgs), you usually do not just replace your work by theirs and then propagate updates in the same way you would have done if you trusted them as much as you. You usually update less hard because the fact that they are not as trusted as you is priced-in. And so the sort of argument tree with mistakes in them causing downstream mistakes that this paper presents is not how I expect it to look like in practice.
In practice (in the sort of Slopolis world that this paper focuses on), I expect humans to notice some slop, know that slop exists, but not be able to know exactly where it is, and thus update much less hard on AI-generated results than on results in which they were heavily involved, similar to
how you would update less hard on the good results of a fancy technique in an academic paper because you know that the actual results are often less good than they seem
how you would update less on your own results if you did them sloppily
how people already don’t trust AI-generated code that much and ask for stronger evidence about code correctness than if a human wrote it
While this can result in some miscalibration due to confirmation bias (which can result in people updating on bad evidence), I think the main effect this will have is a large degradation in effective productivity. I expect it to look something like “you have millions of AI-generated paper-size contributions that look as good as human-generated ones, but you don’t trust them that much because they are AI-generated, and so the size of your update away from your priors is only as good as if you had thousands of paper-size human contributions”.
This also means that I expect people to stay closer to their priors than what you might have hoped based on the volume of research that will happen. And so if you have different priors from decision makers, it might feel like they are miscalibrated, when in fact most of the effect size may just be that they had different priors from you and they (correctly) didn’t update that much given that the evidence was bad.
Thanks for the comment! I think bad evidence, or the difficulty of knowing how much to trust AI-generated research is probably a core issue with automating the work.
I wonder how much room there is to increase the signal:noise ratio of AI-generated work by building a good interface (such that even if you are still updating less than on the equivalent volume of work from humans, you still get good value-for-money from running lots of parallel agents). I’m assuming a world where lots of (empirical) alignment work can be done by agents; in that world, human researchers are the bottleneck. How do you help those researchers make sense of the work done by agents?
I’ve been prototyping this, but so far I haven’t found a version that I personally find useful. It seems to devolve into imposing a structure: a research “contract” that specifies the hypothesis and work to be delivered, artefacts which have a 1-click replication script, an interface that displays all this together with any shared assumptions (to potentially discount correlated evidence) or resources (e.g. if 3 workstreams use the same dataset, if there is something off about that dataset all 3 are invalidated), and so on. I’m imagining easy replication via other agents is also useful, as well as any other work that would lift the ceiling of what a person can reasonably oversee (pairing the main worker with a “falsifier” agent, LLM as a judge hooked in at various points...).
I think some of these could work. That said, I have a high bar for replacing my default agent-assisted workflow, and this hasn’t cleared it yet. I’m curious if you think there is (and if so, where) value in improving the interaction surface.