I did some research in black-box AI control last summer (watch this space, the paper still isn’t out yet) where we called this problem “attack selection” (because the AI is only attacking when the time is right). We (roughly) ended up concluding that there are three really big questions in the textbook untrusted monitoring setup, and only one of them (measuring how good your AI is at recognising its own attacks, as opposed to honeypots) has really been addressed in the major control papers. The other two problems are what to do when you catch your AI doing something bad, which has been discussed informally in Redwood’s blogposts but mostly, and how to measure your AI’s attack selection, which basically hasn’t been studied.
I did some research in black-box AI control last summer (watch this space, the paper still isn’t out yet) where we called this problem “attack selection” (because the AI is only attacking when the time is right). We (roughly) ended up concluding that there are three really big questions in the textbook untrusted monitoring setup, and only one of them (measuring how good your AI is at recognising its own attacks, as opposed to honeypots) has really been addressed in the major control papers. The other two problems are what to do when you catch your AI doing something bad, which has been discussed informally in Redwood’s blogposts but mostly, and how to measure your AI’s attack selection, which basically hasn’t been studied.