AI control is supposed to defend against human-level AI, but the line between a human red team and an AI-augmented one has started to blur. And we’re still optimizing our proxy task of safety-usefulness frontiers as if that line were clean.
What in the control setup relies on a non-AI-augmented red team, and what breaks when you relax this assumption?
Thanks! Good call on making this line of reasoning more explicit.
Your attack policy is supposed to give evidence of a bound on what a real scheming model can do. Assume the attack policy is written by a non-AI-augmented red team. Under that assumption the attack policy is still auditable. You can look at it and say “This attack will work for XY reasons” and test your hypothesis against the monitor. When you relax this constraint and let AI generate the attack policy, it can look auditable, but the provenance is broken — you can reason about what makes it work, but the reasoning is ad-hoc, which is bad epistemic practice.
On a different but related note: An untrusted model doing side-tasks implicitly assumes that the untrusted model will perform its’ side-task to the best of its’ abilities. There’s implicit trust to not scheme, i.e sandbag; do the side-tasks poorly and get caught. If a schemer is testing your control protocols and generating your attack policies, you are kinda doomed.
What in the control setup relies on a non-AI-augmented red team, and what breaks when you relax this assumption?
Thanks! Good call on making this line of reasoning more explicit.
Your attack policy is supposed to give evidence of a bound on what a real scheming model can do. Assume the attack policy is written by a non-AI-augmented red team. Under that assumption the attack policy is still auditable. You can look at it and say “This attack will work for XY reasons” and test your hypothesis against the monitor. When you relax this constraint and let AI generate the attack policy, it can look auditable, but the provenance is broken — you can reason about what makes it work, but the reasoning is ad-hoc, which is bad epistemic practice.
On a different but related note: An untrusted model doing side-tasks implicitly assumes that the untrusted model will perform its’ side-task to the best of its’ abilities. There’s implicit trust to not scheme, i.e sandbag; do the side-tasks poorly and get caught. If a schemer is testing your control protocols and generating your attack policies, you are kinda doomed.