Let me make a simpler form of this problem. Suppose I flip a fair coin a thousand times, and it just happens to land on heads every time. How do I find out that this is a fair coin, and that I don’t actually have a trick coin that always lands on heads? The answer is that I can’t. Any algorithm that tells me that it’s fair is going to fail in the much more likely circumstance that I have a coin that always lands on heads. The best I can do is show that I have 1000 bits of evidence in favor of a trick coin, update my priors accordingly, and use this information when betting.
The good news is that you will only get a coin that lands on heads a thousand times about 00.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 0000000000000000000933% of the time, so you won’t be this wrong by chance very often. In general, you can calculate how likely you are to be wrong, and hedge your bets accordingly.
Let me make a simpler form of this problem. Suppose I flip a fair coin a thousand times, and it just happens to land on heads every time. How do I find out that this is a fair coin, and that I don’t actually have a trick coin that always lands on heads? The answer is that I can’t. Any algorithm that tells me that it’s fair is going to fail in the much more likely circumstance that I have a coin that always lands on heads. The best I can do is show that I have 1000 bits of evidence in favor of a trick coin, update my priors accordingly, and use this information when betting.
The good news is that you will only get a coin that lands on heads a thousand times about 00.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 0000000000000000000933% of the time, so you won’t be this wrong by chance very often. In general, you can calculate how likely you are to be wrong, and hedge your bets accordingly.