I would love there to exist something like “CleanupBench,” if it doesn’t already:
Imagine that a rogue agent has set certain bad computing actions into motion. How well can a different agent identify and shut down what was set into motion? How well can it undo the effects of the actions?
Maybe this is kind of trivial if the cleanup agent has logs of what actions the bad-agent took, and the ability to inspect the underlying programs that were run? I’d be interested in that being tested though.
For context, my organization Guidelight has standards for AI control, which relate to wanting companies to measure how quickly they can achieve a shutdown of a misaligned agent, and to experiment with being able to backtrace through bad actions taken by its agents.
In terms of possibly related stuff: Claude flagged OpenSec as somewhat-but-not-completely relevant, essentially a benchmark for incident response agents. Claude also flagged StepShield, which is about ‘at what step of a bad trajectory is a violation noticed’, though I think that’s less relevant-seeming than OpenSec.
I would love there to exist something like “CleanupBench,” if it doesn’t already:
Imagine that a rogue agent has set certain bad computing actions into motion. How well can a different agent identify and shut down what was set into motion? How well can it undo the effects of the actions?
Maybe this is kind of trivial if the cleanup agent has logs of what actions the bad-agent took, and the ability to inspect the underlying programs that were run? I’d be interested in that being tested though.
For context, my organization Guidelight has standards for AI control, which relate to wanting companies to measure how quickly they can achieve a shutdown of a misaligned agent, and to experiment with being able to backtrace through bad actions taken by its agents.
In terms of possibly related stuff: Claude flagged OpenSec as somewhat-but-not-completely relevant, essentially a benchmark for incident response agents. Claude also flagged StepShield, which is about ‘at what step of a bad trajectory is a violation noticed’, though I think that’s less relevant-seeming than OpenSec.