Here’s a recent article by Andrew Gelman on how easy it is to find and publish completely meaningless “significant” results.
The key concept here is “researcher degrees of freedom”: all of the things the researcher could have tested, but didn’t, or did, but didn’t bother to mention as there was no p<0.05 result.
The money quote (albeit buried at the very end of the article):
Even if something is published in the flagship journal of the leading association of research psychologists, there’s no reason to believe it. The system of scientific publication is set up to encourage publication of spurious findings.
This is strong stuff too:
Researchers’ decisions about which variables to analyze may make perfect sense, but they indicate the difficulty of taking these p-values at anything like face value. There is no reason to assume the researchers were doing anything nefarious or trying to manipulate the truth. Rather, like sculptors, they were chipping away the pieces of the data that did not fit their story, until they ended up with a beautiful and statistically significant structure that confirmed their views.
On his blog he calls this “the scientific mass production of spurious statistical significance”.
If you assume regression to the mean will occur in replication, you should only pay attention to fairly large effect sizes. This has a way of washing out a lot of the noise in areas where previously it seemed hard to draw a conclusion.
Here’s a recent article by Andrew Gelman on how easy it is to find and publish completely meaningless “significant” results.
The key concept here is “researcher degrees of freedom”: all of the things the researcher could have tested, but didn’t, or did, but didn’t bother to mention as there was no p<0.05 result.
The money quote (albeit buried at the very end of the article):
This is strong stuff too:
On his blog he calls this “the scientific mass production of spurious statistical significance”.
If you assume regression to the mean will occur in replication, you should only pay attention to fairly large effect sizes. This has a way of washing out a lot of the noise in areas where previously it seemed hard to draw a conclusion.