“Five out of six epidemiological studies have been contradicted in a very short period of time,” says Ioannidis, “while about one out of three randomized clinical trials were also refuted.”
Note that this is not 1⁄3 of all randomized studies, but just the highly cited ones, the ones that have any chance of an attempt at replication (and still 25% have no published attempt). It is not obvious to me whether these studies will be higher or lower quality than average. ETA: actually, there is a “control group” of articles published in the same prestigious journals, but with fewer citations. They are contradicted a little less often, p<.1, and not because people aren’t bothering to replicate them.
But there still quite a difference between claiming that 1⁄3 or 2⁄3 of randomized studies are wrong.
Yes, the 2⁄3 is probably an error on Eliezer’s part. He doesn’t say randomized, so he could be misremembering the 5⁄6, but the 5⁄6 is probably not the figure to quote and Ioannidis is a little misleading in putting it first. Only 6 of the 45 highly-cited studies were retrospective and I think that is representative of medical research in the prestigious journals. (the discussion of the control group should address this, but I don’t see it)
In fact, the origin of the statistic is an earlier paper by the same author, John PA Ioannidis, Contradicted and Initially Stronger Effects in Highly Cited Clinical Research. Journalistic coverage of the later paper quotes a description of the earlier:
Note that this is not 1⁄3 of all randomized studies, but just the highly cited ones, the ones that have any chance of an attempt at replication (and still 25% have no published attempt). It is not obvious to me whether these studies will be higher or lower quality than average. ETA: actually, there is a “control group” of articles published in the same prestigious journals, but with fewer citations. They are contradicted a little less often, p<.1, and not because people aren’t bothering to replicate them.
But there still quite a difference between claiming that 1⁄3 or 2⁄3 of randomized studies are wrong.
Yes, the 2⁄3 is probably an error on Eliezer’s part. He doesn’t say randomized, so he could be misremembering the 5⁄6, but the 5⁄6 is probably not the figure to quote and Ioannidis is a little misleading in putting it first. Only 6 of the 45 highly-cited studies were retrospective and I think that is representative of medical research in the prestigious journals. (the discussion of the control group should address this, but I don’t see it)