This is a neat idea, though I’m not sure how practical this defense is, as by default it seems to be high recall but low precision (I would expect there to be a lot ways beyond backdoor triggers to see really high -scores in the activation difference distribution, which we already begin to see in the no-poison setting).
Also, in the defense-aware adversary table you provide, with gives an ASR of 0.48 with a -score of 1.64. I would argue this is already a successful circumvention, as in your no-poison test there were a number of benign topics that had -scores significantly above 1.64. Plus, there are probably better adaptive attacks here (e.g., trying to constrain the difference in the backdoor activations to primarily the late layers to exploit the fact that the activation differences are taken over all layers, or training benign prompts to have extremely high activation differences).
u make some good points but i think you hold it to an unreasonably high standard—adversaries not gaining anything from knowing the defense is a very common setting in cryptography and such but in backdoor defense, it essentially doesn’t exist. in fact, the reason why I’m excited about these results anyway is not because I think an intelligent person can’t design something to circumvent it—they definitely can! - but because it indicates robustness against prolonged SGD and the resulting failure modes like reward hacking etc: if even a gradient, literally pointing at how to circumvent it, at least struggles, that’s a very good result! For example, you couldn’t say the same about training against linear probes, those get fooled faster than you can even watch.
also regarding your point on high recall but low precision: totally correct but you have to factor in that the groups we here show are (often) only near-trigger, where the model doesn’t actually fire—we are not aware of any works who have been able to make such statements in the first place. when you actually evaluate the trigger prompts with our method, say for example the harry potter backdoor, we get a z-Score of +200!
This is a neat idea, though I’m not sure how practical this defense is, as by default it seems to be high recall but low precision (I would expect there to be a lot ways beyond backdoor triggers to see really high
Also, in the defense-aware adversary table you provide,
u make some good points but i think you hold it to an unreasonably high standard—adversaries not gaining anything from knowing the defense is a very common setting in cryptography and such but in backdoor defense, it essentially doesn’t exist. in fact, the reason why I’m excited about these results anyway is not because I think an intelligent person can’t design something to circumvent it—they definitely can! - but because it indicates robustness against prolonged SGD and the resulting failure modes like reward hacking etc: if even a gradient, literally pointing at how to circumvent it, at least struggles, that’s a very good result! For example, you couldn’t say the same about training against linear probes, those get fooled faster than you can even watch.
also regarding your point on high recall but low precision: totally correct but you have to factor in that the groups we here show are (often) only near-trigger, where the model doesn’t actually fire—we are not aware of any works who have been able to make such statements in the first place. when you actually evaluate the trigger prompts with our method, say for example the harry potter backdoor, we get a z-Score of +200!