This question may be stupid but aren’t canary strings quite fragile in terms safety mechanism for training? If someone were to just copy and paste material from one webpage to another without the string (be it intentionally or not), wouldn’t this defeat the purpose?
If so, finding something a bit more failsafe for researchers to use that avoid contamination seems like a interesting project that would benefit the field.
This doesn’t solve the issue of models figuring out for themselves how to bypass this, but does stop a lot of easy cases of missuses of the information.
I agree that it’s very fragile for certain uses. I think usually the reasoning is something like “this is an informal norm that labs should follow, and which is good if generally enforced, and provides an easy way to detect if not”. There are tons of ways to beat this if you’re adversarially trying to. I agree that finding better ways to avoid contamination would be good—@TurnTrout has a bounty for this, and there are some people are working on this now.
This question may be stupid but aren’t canary strings quite fragile in terms safety mechanism for training? If someone were to just copy and paste material from one webpage to another without the string (be it intentionally or not), wouldn’t this defeat the purpose?
If so, finding something a bit more failsafe for researchers to use that avoid contamination seems like a interesting project that would benefit the field.
This doesn’t solve the issue of models figuring out for themselves how to bypass this, but does stop a lot of easy cases of missuses of the information.
I agree that it’s very fragile for certain uses. I think usually the reasoning is something like “this is an informal norm that labs should follow, and which is good if generally enforced, and provides an easy way to detect if not”. There are tons of ways to beat this if you’re adversarially trying to. I agree that finding better ways to avoid contamination would be good—@TurnTrout has a bounty for this, and there are some people are working on this now.