That’s right—you still only get a bound on average quality, and you need to do something to cope with failures so rare they never appear in training (here’s a post reviewing my best guesses).
But before you weren’t even in the game, it wouldn’t matter how well adversarial training worked because you didn’t even have the knowledge to tell whether a given behavior is good or bad. You weren’t even getting the right behavior on average.
(In the OP I think the claim “the generalization is now coming entirely from human beliefs” is fine, I meant generalization from one distribution to another. “Neural nets are are fine” was sweeping these issues under the rug. Though note that in the real world the distribution will change from neural net training to deployment, it’s just exactly the normal robustness problem. The point of this post is just to get it down to only a robustness problem that you could solve with some kind of generalization of adversarial training, the reason to set it up as in the OP was to make the issue more clear.)
That’s right—you still only get a bound on average quality, and you need to do something to cope with failures so rare they never appear in training (here’s a post reviewing my best guesses).
But before you weren’t even in the game, it wouldn’t matter how well adversarial training worked because you didn’t even have the knowledge to tell whether a given behavior is good or bad. You weren’t even getting the right behavior on average.
(In the OP I think the claim “the generalization is now coming entirely from human beliefs” is fine, I meant generalization from one distribution to another. “Neural nets are are fine” was sweeping these issues under the rug. Though note that in the real world the distribution will change from neural net training to deployment, it’s just exactly the normal robustness problem. The point of this post is just to get it down to only a robustness problem that you could solve with some kind of generalization of adversarial training, the reason to set it up as in the OP was to make the issue more clear.)