Not hundred percent sure I’m parsing you correctly. Given capable AI, the attacker can implant the backdoor trigger during inference whenever possible and rely on the AI to attack select. I’m mostly modeling the attacker as caring about being able to shift the AI propensities to what they desire during inference. I think the attack selection problem looks reasonably similar to that where the AI became misaligned on its own.
I’d like to be able to compute “leverage”—if an attacker has access to random 1% of training episodes and random 5% of inference episodes, how much “total effectiveness” can they recover (where total effectiveness is something like how much of some desired impact they’d achieve with 100% control of both).
The naive computation would say something like: 1% compromise gets you a reliable backdoor, and you then control 5% of inference but you can’t choose it so you get 5% effectiveness. Can you do much better?
I’m not expecting you to have a solid answer to this, but because I’m interested in it I was wondering if you had any thoughts on the question.
Not hundred percent sure I’m parsing you correctly.
Given capable AI, the attacker can implant the backdoor trigger during inference whenever possible and rely on the AI to attack select. I’m mostly modeling the attacker as caring about being able to shift the AI propensities to what they desire during inference. I think the attack selection problem looks reasonably similar to that where the AI became misaligned on its own.
I’d like to be able to compute “leverage”—if an attacker has access to random 1% of training episodes and random 5% of inference episodes, how much “total effectiveness” can they recover (where total effectiveness is something like how much of some desired impact they’d achieve with 100% control of both).
The naive computation would say something like: 1% compromise gets you a reliable backdoor, and you then control 5% of inference but you can’t choose it so you get 5% effectiveness. Can you do much better?
I’m not expecting you to have a solid answer to this, but because I’m interested in it I was wondering if you had any thoughts on the question.