Cool work but something concerns me. The method only scores , so anything the two might share is invisible. The “clean reference” assumption is doing a lot of work here and only gets a weak check.
The Qwen censorship case is a nice lens because the reference’s status flips the failure: if the reference censors and the suspect doesn’t, you get a false positive spike on China-sensitive prompts. If both censor, the difference cancels and you get a false negative, the shared behavior just gets subtracted away.
My first instinct for a fix was to use an ensemble of cross-family references instead of a single one, and only trust a residual spike that shows up across several of them. But that half-works at best, and I don’t think one rule covers both failures. Asking for a spike in many references removes the false positives, because a quirk of one single reference will not show up in the others. But for shared behavior it does the opposite: here the clean references are the ones that spike, while the contaminated ones stay flat, so asking for agreement hides the true positive. And either way, if the behavior is inherited from a shared base or is ecosystem-wide, every reference carries it and the whole ensemble stays blind.
Does this track? Curious whether you considered contaminated references at all.
Cool work but something concerns me. The method only scores
The Qwen censorship case is a nice lens because the reference’s status flips the failure: if the reference censors and the suspect doesn’t, you get a false positive spike on China-sensitive prompts. If both censor, the difference cancels and you get a false negative, the shared behavior just gets subtracted away.
My first instinct for a fix was to use an ensemble of cross-family references instead of a single one, and only trust a residual spike that shows up across several of them. But that half-works at best, and I don’t think one rule covers both failures. Asking for a spike in many references removes the false positives, because a quirk of one single reference will not show up in the others. But for shared behavior it does the opposite: here the clean references are the ones that spike, while the contaminated ones stay flat, so asking for agreement hides the true positive. And either way, if the behavior is inherited from a shared base or is ecosystem-wide, every reference carries it and the whole ensemble stays blind.
Does this track? Curious whether you considered contaminated references at all.