Question: Would a proposal be ruled out by a counterexample even if that counterexample is exponentially unlikely?
I’m imagining a theorem, proved using some large deviation estimate, of the form: If the model satisfies hypotheses XYZ, then it is exponentially unlikely to learn W. Exponential in the number of parameters, say. In which case, we could train models like this until the end of the universe and be confident that we will never see a single instance of learning W.
Question: Would a proposal be ruled out by a counterexample even if that counterexample is exponentially unlikely?
I’m imagining a theorem, proved using some large deviation estimate, of the form: If the model satisfies hypotheses XYZ, then it is exponentially unlikely to learn W. Exponential in the number of parameters, say. In which case, we could train models like this until the end of the universe and be confident that we will never see a single instance of learning W.
I’d be fine with a proposal that flips coins and fails with small probability (in every possible world).