u make some good points but i think you hold it to an unreasonably high standard—adversaries not gaining anything from knowing the defense is a very common setting in cryptography and such but in backdoor defense, it essentially doesn’t exist. in fact, the reason why I’m excited about these results anyway is not because I think an intelligent person can’t design something to circumvent it—they definitely can! - but because it indicates robustness against prolonged SGD and the resulting failure modes like reward hacking etc: if even a gradient, literally pointing at how to circumvent it, at least struggles, that’s a very good result! For example, you couldn’t say the same about training against linear probes, those get fooled faster than you can even watch.
also regarding your point on high recall but low precision: totally correct but you have to factor in that the groups we here show are (often) only near-trigger, where the model doesn’t actually fire—we are not aware of any works who have been able to make such statements in the first place. when you actually evaluate the trigger prompts with our method, say for example the harry potter backdoor, we get a z-Score of +200!
u make some good points but i think you hold it to an unreasonably high standard—adversaries not gaining anything from knowing the defense is a very common setting in cryptography and such but in backdoor defense, it essentially doesn’t exist. in fact, the reason why I’m excited about these results anyway is not because I think an intelligent person can’t design something to circumvent it—they definitely can! - but because it indicates robustness against prolonged SGD and the resulting failure modes like reward hacking etc: if even a gradient, literally pointing at how to circumvent it, at least struggles, that’s a very good result! For example, you couldn’t say the same about training against linear probes, those get fooled faster than you can even watch.
also regarding your point on high recall but low precision: totally correct but you have to factor in that the groups we here show are (often) only near-trigger, where the model doesn’t actually fire—we are not aware of any works who have been able to make such statements in the first place. when you actually evaluate the trigger prompts with our method, say for example the harry potter backdoor, we get a z-Score of +200!