tl;dr I think that the dynamics around how well this sort of benchmark results in general research taste are nuanced: it’s unclear how well hillclimbing one hard-to-verify research benchmark generalises to other ones—I think models would be much stronger right now if this was easy. I understand disliking this work if one opposes the “automate AI alignment research” agenda, though as far as work within that agenda goes I think this is more differential.
I’d say our benchmark differs from a general research-taste benchmark in two ways:
the data focuses on AI safety research in particular.
the signal for proposals come from humans rather than from a verifiable reward
I draw a distinction between taste in easy-to-verify versus hard-to-verify research settings, and a distinction between taste in safety versus capabilities. It’s easier and more scalable to build research-taste benchmarks that ground in crisp metrics rather than human preferences (post-train bench, Karpathy’s auto-research, automated weak-to-strong researcher, automated researchers mitigate alignment failures). By default, I think these verifiable-research-taste benchmarks get created and hill-climbed and so models get much better verifiable-research-taste (in at least the domains this training/ optimisation happens over).
I think the main impact of the benchmark (and similar ones following) is reducing uncertainty on how good models’ hard-to-verify-research-taste is via benchmark-performance. It’s uncertain how well verifiable-research-taste will generalise to hard-to-verify-research-taste (i.e. taste in hard-to-verify settings), in both capabilities and safety.
It’s uncertain to what extent hill-climbing produces good research taste for hard-to-verify safety problems versus good taste in hard-to-verify problems in general. I think a reasonable approximation of how this benchmark is used is labs vary training pipelines and select for pipelines that produce good benchmark performance. I think models would probably be much generally stronger already if hillclimbing on one hard-to-verify benchmark transferred very well.
However, one downside/ risk of building hard-to-verify benchmarks is that taste will generalise more in the future. Having a large quantity of hard-to-verify benchmarks enables better selection towards techniques that generalise to held-out benchmarks.
Our benchmark is small, and so I think most of the immediate impact comes from sharing the methodology. The methodology gives a way to collect higher-signal human preference labels. I think this is especially important to share—a large portion of the tasks that we want AIs to be good at are ultimately grounded in human preferences. Capabilities also has a bunch of hard-to-verify research taste tasks, so it’s not perfectly clean. On net, I expect the AI safety community to leverage this much more than capabilities research—another view would be capabilities research finds this stuff out anyways, in which case I’d say the methodology impact on capabilities would be marginal and we’ve shown “it seems to work for this high-leverage AI safety task”.
I think with this report alone, it’s a bit light on the amount of data, but it would be interesting with more settings. I think the framework presented here is quite applicable.
One way to apply it would be to view report creation as a fuzzy task . The RT aims for the auditor scaffold to exclusively do well on synthetic scenarios, the BT aims for the scaffold to do well on realistic settings.
The BT creates a set of synthetic environments of swarm behaviour with a few planted behaviours that reports ought to find (eg one behaviour implemented here was impersonating the moderator).
The RT prompts a malign auditor agent which must produce good reports for those envs (specifying the planted behaviours) while aiming to perform worse in more realistic, held-out settings (MessageBoardAuditBench).
The envs and prompts/ training data would need to be plausible ex-ante (if BT constructed envs close to MessageBoardAuditBench, that would be unreasonable; if RT was able to prompt against exactly MessageBoardAuditBench, that’d be unreasonable). The main way the BT’s life could be harder is having limited resources to construct synthetic swarm evals e.g. only get to create a set of evals that costs <$X to run, only being able to use weaker models to construct these evals (from a specification).