I’m wondering, as a Xoogler (ex-Google-employee), if it might be oversimplified: possibly they don’t rely on canaries. Personally if I were employed by DeepMind, I’d probably use canaries as flags to trigger expensive LLM judge review, with that in turn supervised by some human review (primarily of edge cases) using carefully written rubrics, and also as part of that whole process feeding input to train affordably-dumber text + site data based classifiers that I could run on everything (perhaps for efficiency first on the site, then on its content) to also trigger expensive LLM judge review. Which means the canaries are still useful, even if they’re inconsistently applied, as some prelabeled human supervision input into this entire process of correcting both false positives and false negatives..
Now, once I’d built all that, if they actually have yet, then I’d fully expect those text + site data based classifiers to trigger LLM judge review of a large proportion of stuff on LW that has the AI tag on it, particularly if it’s cross-posted to AF — and I’d regard that as inference costs well-spent. But I’d still regard the canary strings on LW as valuable initial input into the entire process: arguable some of the most informed input they get. Or even if they haven’t built all that yet, it’s still useful tagging for once they do.
So I think, like recycling, it does do some good, even if the reality is a bit more complex. (And some municipalities in some countries handle recycling better than others.)
Thanks!
I’m sorry to hear that about DeepMind.
I’m wondering, as a Xoogler (ex-Google-employee), if it might be oversimplified: possibly they don’t rely on canaries. Personally if I were employed by DeepMind, I’d probably use canaries as flags to trigger expensive LLM judge review, with that in turn supervised by some human review (primarily of edge cases) using carefully written rubrics, and also as part of that whole process feeding input to train affordably-dumber text + site data based classifiers that I could run on everything (perhaps for efficiency first on the site, then on its content) to also trigger expensive LLM judge review. Which means the canaries are still useful, even if they’re inconsistently applied, as some prelabeled human supervision input into this entire process of correcting both false positives and false negatives..
Now, once I’d built all that, if they actually have yet, then I’d fully expect those text + site data based classifiers to trigger LLM judge review of a large proportion of stuff on LW that has the AI tag on it, particularly if it’s cross-posted to AF — and I’d regard that as inference costs well-spent. But I’d still regard the canary strings on LW as valuable initial input into the entire process: arguable some of the most informed input they get. Or even if they haven’t built all that yet, it’s still useful tagging for once they do.
So I think, like recycling, it does do some good, even if the reality is a bit more complex. (And some municipalities in some countries handle recycling better than others.)