Note: I currently think that the basic picture of getting within ϵ of a good prediction is actually pretty sketchy. I wrote about the sample complexity here. Additional to the sample complexity issues, the requirement is for predictors to be Bayes-optimal, but Bayes-optimality is not possible for bounded reasoners. This is important because e.g. some adversarial predictor might make very good predictions on some subset of questions (because it’s spending its compute on those specifically), causing other predictors to be filtered out (if those questions are used to determine who the best predictor is). I don’t know what kind of analysis could get the ϵ-accuracy result at this point.
Note: I currently think that the basic picture of getting within ϵ of a good prediction is actually pretty sketchy. I wrote about the sample complexity here. Additional to the sample complexity issues, the requirement is for predictors to be Bayes-optimal, but Bayes-optimality is not possible for bounded reasoners. This is important because e.g. some adversarial predictor might make very good predictions on some subset of questions (because it’s spending its compute on those specifically), causing other predictors to be filtered out (if those questions are used to determine who the best predictor is). I don’t know what kind of analysis could get the ϵ-accuracy result at this point.