There should always, really, be “allowance in the original problem”. Perhaps not explicitly factored in, but you should assign some nonzero probability to possibilities like “the experimenter lied to me”, “I goofed in some crazy way”, “I am being deceived by malevolent demons”, etc. In practice, these wacky hypotheses may not occur to you until the evidence for them starts getting large, and you can decide at that point what prior probabilities you should have put on them. (Unfortunately it’s easy to do that wrongly, e.g. because of hindsight bias.)
As Douglas_Knight says, frequentist statistics is full of tests that will tell you when some otherwise plausible hypothesis (e.g., “these two samples are drawn from things with the same probability distribution”) are incompatible with the data in particular (or not-so-particular) ways.
There should always, really, be “allowance in the original problem”. Perhaps not explicitly factored in, but you should assign some nonzero probability to possibilities like “the experimenter lied to me”, “I goofed in some crazy way”, “I am being deceived by malevolent demons”, etc. In practice, these wacky hypotheses may not occur to you until the evidence for them starts getting large, and you can decide at that point what prior probabilities you should have put on them. (Unfortunately it’s easy to do that wrongly, e.g. because of hindsight bias.)
As Douglas_Knight says, frequentist statistics is full of tests that will tell you when some otherwise plausible hypothesis (e.g., “these two samples are drawn from things with the same probability distribution”) are incompatible with the data in particular (or not-so-particular) ways.