If you say you’ve proved hypothesis H, but you don’t do the statistics right, then that means there’s some possibility E, distinct from H, that can account for your results. I call it E because it’s the possibility in which the conclusion is an error.
Therefore, disbelieving the study is the same as believing E. But E might be just as unlikely as H, which is something you’ve got to consider before rejecting the study.
It gets even worse for you if they’ve got 10 studies confirming H, with decorrelated error. What that means is that H accounts for all of them (that’s why they’re all studies confirming H), and the “decorrelated error” part means there’s no single possibility E in which all their bad statistical analyses would fail. Instead, there’s E1, E2, E3...
At that point, the chance that E1 through E10 are all true is way less likely than H.
This is part of why different kinds of evidence are important: to decorrelate the error in imperfect analyses.
An example: the criticism that a conclusion H is based on studies of WEIRD subjects (Western, Educated, Industrialized, Rich, and Democratic) is serious, because it’s a single circumstance E that accounts for all the studies. The errors are correlated. However, you’ve still got to consider, is the hypothesis E, “H is true for WEIRD people”, more likely than what the scientists believe, that H is true for everybody? Disbelieving the conclusion because the subjects are WEIRD still commits you to a pretty specific belief.
There are other possible sources of correlation, e.g., scientists playing around with the statistics until they get a result that agrees what they expect.
Going a little farther.
If you say you’ve proved hypothesis H, but you don’t do the statistics right, then that means there’s some possibility E, distinct from H, that can account for your results. I call it E because it’s the possibility in which the conclusion is an error.
Therefore, disbelieving the study is the same as believing E. But E might be just as unlikely as H, which is something you’ve got to consider before rejecting the study.
It gets even worse for you if they’ve got 10 studies confirming H, with decorrelated error. What that means is that H accounts for all of them (that’s why they’re all studies confirming H), and the “decorrelated error” part means there’s no single possibility E in which all their bad statistical analyses would fail. Instead, there’s E1, E2, E3...
At that point, the chance that E1 through E10 are all true is way less likely than H.
This is part of why different kinds of evidence are important: to decorrelate the error in imperfect analyses.
An example: the criticism that a conclusion H is based on studies of WEIRD subjects (Western, Educated, Industrialized, Rich, and Democratic) is serious, because it’s a single circumstance E that accounts for all the studies. The errors are correlated. However, you’ve still got to consider, is the hypothesis E, “H is true for WEIRD people”, more likely than what the scientists believe, that H is true for everybody? Disbelieving the conclusion because the subjects are WEIRD still commits you to a pretty specific belief.
There are other possible sources of correlation, e.g., scientists playing around with the statistics until they get a result that agrees what they expect.