That’s very interesting but it’s about replicability not fraud and those are (at least sometimes) very different things.
Possible counters:
1. “Not so different: results that don’t replicate are probably fraudulent.” I’m prepared to be persuaded but I’d be surprised if most reproducibility-failures were the result of fraud.
2. “Whatever differences between ‘worse’ and ‘better’ publications you have in mind that you’d hope would reduce fraud in the ‘better’ ones, you should also expect to reduce nonreplicability; apparently they don’t do that, so why expect them to reduce fraud?” My impression is that a lot of nonreplicability is just “lucky”: you happened to get a result with p<0.05 and so you published it, and if you’d got p>0.05 maybe you wouldn’t have bothered, or maybe the fancy journal wouldn’t have been interested and it would have been published somewhere worse. This mechanism will lead to nonreplicable papers being higher-profile and cited more, without there being anything much about those papers that would raise red flags if a journal is worried about fraud. So to whatever extent fraud is detectable in advance, and better journals try harder to spot it because their reputation is more valuable, better journals will tend to see less fraud but not less nonreplicability. (And: to whatever extent fraudsters expect better journals to try harder to catch fraud, they will tend to avoid publishing their fraudulent papers there.)
I like your second argument. But to be honest, there is a giant grey area between “non-replicable” and “fraudulent”. It is hard to draw the line between, “Intellectually dishonest but didn’t mean to deceive” and “fraudulent”. And even if you could define the line, we lack the data to identify what falls on either side.
It is worth reading Time to assume that health research is fraudulent until proven otherwise? I believe that this case is an exception to Betteridge’s law—I think that the answer is yes. Given the extraordinary efforts that the editor of Anaesthesia needed to catch some fraud, I doubt that most journals do it. And because I believe that, I’m inclined to a prior that says that non-replicability suggests at least even odds of fraud.
As a sanity check, high profile examples like the former President of Stanford demonstrate that fraudulent research is accepted in top journals, leading to prestigious positions. See also the case of Dr. Francesca Gino, formerly of Harvard.
And, finally, back to the line between intellectual dishonesty and fraud. I’m inclined to say that they amount to the same thing in practice, and we should treat them similarly. And the combined bucket is a pretty big problem.
Here is a good example. The Cargo Cult Science speech happened around 50 years ago. Psychologists have objected ever since to being called a pseudoscience by many physicists. But it took 40 years before they finally did what Feynman told them to, and tried replicating their results. They generally have not acknowledged Feynman’s point, nor have the started fixing the other problems that Feynman talked about.
Given that, how much faith should we put in psychology?
The effect isn’t large, but you’d lose that bet. See Nonreplicable publications are cited more than replicable ones.
That’s very interesting but it’s about replicability not fraud and those are (at least sometimes) very different things.
Possible counters:
1. “Not so different: results that don’t replicate are probably fraudulent.” I’m prepared to be persuaded but I’d be surprised if most reproducibility-failures were the result of fraud.
2. “Whatever differences between ‘worse’ and ‘better’ publications you have in mind that you’d hope would reduce fraud in the ‘better’ ones, you should also expect to reduce nonreplicability; apparently they don’t do that, so why expect them to reduce fraud?” My impression is that a lot of nonreplicability is just “lucky”: you happened to get a result with p<0.05 and so you published it, and if you’d got p>0.05 maybe you wouldn’t have bothered, or maybe the fancy journal wouldn’t have been interested and it would have been published somewhere worse. This mechanism will lead to nonreplicable papers being higher-profile and cited more, without there being anything much about those papers that would raise red flags if a journal is worried about fraud. So to whatever extent fraud is detectable in advance, and better journals try harder to spot it because their reputation is more valuable, better journals will tend to see less fraud but not less nonreplicability. (And: to whatever extent fraudsters expect better journals to try harder to catch fraud, they will tend to avoid publishing their fraudulent papers there.)
I like your second argument. But to be honest, there is a giant grey area between “non-replicable” and “fraudulent”. It is hard to draw the line between, “Intellectually dishonest but didn’t mean to deceive” and “fraudulent”. And even if you could define the line, we lack the data to identify what falls on either side.
It is worth reading Time to assume that health research is fraudulent until proven otherwise? I believe that this case is an exception to Betteridge’s law—I think that the answer is yes. Given the extraordinary efforts that the editor of Anaesthesia needed to catch some fraud, I doubt that most journals do it. And because I believe that, I’m inclined to a prior that says that non-replicability suggests at least even odds of fraud.
As a sanity check, high profile examples like the former President of Stanford demonstrate that fraudulent research is accepted in top journals, leading to prestigious positions. See also the case of Dr. Francesca Gino, formerly of Harvard.
And, finally, back to the line between intellectual dishonesty and fraud. I’m inclined to say that they amount to the same thing in practice, and we should treat them similarly. And the combined bucket is a pretty big problem.
Here is a good example. The Cargo Cult Science speech happened around 50 years ago. Psychologists have objected ever since to being called a pseudoscience by many physicists. But it took 40 years before they finally did what Feynman told them to, and tried replicating their results. They generally have not acknowledged Feynman’s point, nor have the started fixing the other problems that Feynman talked about.
Given that, how much faith should we put in psychology?