“Statistically significant results” mean that there’s a 5% chance that results are wrong
Wrong. It means that the researcher defined a class of results such that the class had less than a 5% chance of occurring if the null hypothesis were true, and that the actual outcome fell into this class.
There are all sorts of things that can go wrong with that, but, even leaving all those aside, it doesn’t mean there’s a 5% chance the results are wrong. Suppose you’re investigating psychic powers, and that the journals have (as is usually the case!) a heavy publication bias toward positive results. Then the journal will be full of statistically significant results and they will all be wrong.
In fairness, your last point isn’t really about confidence levels. A journal that only accepted papers written in the Bayesian methodology, but had the same publication bias, would be just as wrong.
A journal that reported likelihood ratios would at least be doing better.
A journal that actually cared about science would accept papers before the experiment had been done, with a fixed statistical methodology submitted with the paper in advance rather than data-mining the statistical significance afterward.
I’m confused by your remark. “5% chance of false positive” obviously means P(positive results|null hypothesis true)=5%, P(null hypothesis true|positive results) is subjective and has no particular meaningful value, so I couldn’t have talked about that.
Wrong. It means that the researcher defined a class of results such that the class had less than a 5% chance of occurring if the null hypothesis were true, and that the actual outcome fell into this class.
There are all sorts of things that can go wrong with that, but, even leaving all those aside, it doesn’t mean there’s a 5% chance the results are wrong. Suppose you’re investigating psychic powers, and that the journals have (as is usually the case!) a heavy publication bias toward positive results. Then the journal will be full of statistically significant results and they will all be wrong.
In fairness, your last point isn’t really about confidence levels. A journal that only accepted papers written in the Bayesian methodology, but had the same publication bias, would be just as wrong.
A journal that reported likelihood ratios would at least be doing better.
A journal that actually cared about science would accept papers before the experiment had been done, with a fixed statistical methodology submitted with the paper in advance rather than data-mining the statistical significance afterward.
Is this meant to suggest that journal editors literally don’t care about science that much, or simply that “people are crazy, the world is mad”?
Not an objection, but a lot of the articles in that journal would be “here’s my reproduction of the results I got last year and published then”.
...which is a really good thing, on reflection.
I’m confused by your remark. “5% chance of false positive” obviously means P(positive results|null hypothesis true)=5%, P(null hypothesis true|positive results) is subjective and has no particular meaningful value, so I couldn’t have talked about that.