Hm, so you think if there are some distinctive benchmark questions that have been discussed online, models otherwise trained on that era of internet won’t know details about them?
Probably you should go read my other comment threads on this issue if you want details, but Google’s approach is designed to filter out text that includes benchmark questions regardless of whether there is a canary string. I’m sure it’s not perfect but I think it’s pretty good.
I make no such claims about any other models, just gemini where I have direct knowledge.
Hm, so you think if there are some distinctive benchmark questions that have been discussed online, models otherwise trained on that era of internet won’t know details about them?
Probably you should go read my other comment threads on this issue if you want details, but Google’s approach is designed to filter out text that includes benchmark questions regardless of whether there is a canary string. I’m sure it’s not perfect but I think it’s pretty good.
I make no such claims about any other models, just gemini where I have direct knowledge.