I don’t think that this is as big a deal as people are making it out to be. There’s very little in the system which protects against fraud. Whenever there’s some form of fraud that shows up people make a big deal about it, but the bottom line is that fraud is rare enough that it is likely that protections against fraud would be a poor use of limited resources.
I think people like Andrew Wakefield make the effects of fraud more obvious. And, I agree, fraud on that level is probably rare, but what about smaller acts of fraud? For instance, I don’t think it’s that unlikely that many scientists, while under pressure and deadlines, fudge their results. And not because they want to deceive, no, they already “know” what the results should be, so they’re not doing anything that’s really all that wrong.
Interestingly though, we’ve found that science works despite significant bias, poor research, and so on. So, I wouldn’t be surprised if there was a significant amount of fraud, and yet we still were able to do science.
Basically, I don’t think the question should be “is this really a big deal?” but “how much better would science be if this were fixed?”.
I’ve read a few articles on this issue, and the problem seems pretty alarming. From what I understand, there are only a small handful of journals that accept null hypothesis (i.e. hypothesis that X is not true), in fact I think there might be only one (JASN).
The vast majority of journals reject (or at least discourage) non-positive results except in the case of famous researchers or contentious issues, which means studies that show negative results tend to not get published. In fact, most researchers don’t even attempt to publish—they start over, or give up and move to a different project. If the study was critical to their career, they may even move to another field entirely.
This meta-study examines studies of publication bias and reporting bias (unfavorable results omitted from conclusions). It comes to the conclusion that studies that show positive results are significantly more likely to be published than studies that don’t.
If the academic culture is discouraging studies that show negative results via the publication process, doesn’t that seem to imply there is, at the very least, a major inefficiency in our process of learning new things?
I’m no scientist, but I do have to do a lot of troubleshooting for my job, and while knowing what works is most important, knowing what doesn’t takes a close second.
If journals encouraged negative results as much as positive results, I’d imagine we’d see major new scientific breakthroughs twice as often as we currently do—and that’s kind of a big deal, I think. Right now the huge academic pressure is not to produce valid results, but to produce positive results. That’s a major problem, in my opinion, precisely because human beings are very vulnerable to such pressure. There are a whole slew of biases that arise because of that type of pressure, and any researcher not very cognizant of their vulnerability is another potential Staple.
It seems inexplicably strange to me that journals publish papers from researchers who don’t make all of their data public, except where privacy of participants could be infringed upon.
I don’t think that this is as big a deal as people are making it out to be. There’s very little in the system which protects against fraud. Whenever there’s some form of fraud that shows up people make a big deal about it, but the bottom line is that fraud is rare enough that it is likely that protections against fraud would be a poor use of limited resources.
I think people like Andrew Wakefield make the effects of fraud more obvious. And, I agree, fraud on that level is probably rare, but what about smaller acts of fraud? For instance, I don’t think it’s that unlikely that many scientists, while under pressure and deadlines, fudge their results. And not because they want to deceive, no, they already “know” what the results should be, so they’re not doing anything that’s really all that wrong.
Interestingly though, we’ve found that science works despite significant bias, poor research, and so on. So, I wouldn’t be surprised if there was a significant amount of fraud, and yet we still were able to do science.
Basically, I don’t think the question should be “is this really a big deal?” but “how much better would science be if this were fixed?”.
Opportunity cost is a big deal when it comes to top quality human minds, considering they are so rare.
I’ve read a few articles on this issue, and the problem seems pretty alarming. From what I understand, there are only a small handful of journals that accept null hypothesis (i.e. hypothesis that X is not true), in fact I think there might be only one (JASN).
The vast majority of journals reject (or at least discourage) non-positive results except in the case of famous researchers or contentious issues, which means studies that show negative results tend to not get published. In fact, most researchers don’t even attempt to publish—they start over, or give up and move to a different project. If the study was critical to their career, they may even move to another field entirely.
This meta-study examines studies of publication bias and reporting bias (unfavorable results omitted from conclusions). It comes to the conclusion that studies that show positive results are significantly more likely to be published than studies that don’t.
If the academic culture is discouraging studies that show negative results via the publication process, doesn’t that seem to imply there is, at the very least, a major inefficiency in our process of learning new things?
I’m no scientist, but I do have to do a lot of troubleshooting for my job, and while knowing what works is most important, knowing what doesn’t takes a close second.
If journals encouraged negative results as much as positive results, I’d imagine we’d see major new scientific breakthroughs twice as often as we currently do—and that’s kind of a big deal, I think. Right now the huge academic pressure is not to produce valid results, but to produce positive results. That’s a major problem, in my opinion, precisely because human beings are very vulnerable to such pressure. There are a whole slew of biases that arise because of that type of pressure, and any researcher not very cognizant of their vulnerability is another potential Staple.
It seems inexplicably strange to me that journals publish papers from researchers who don’t make all of their data public, except where privacy of participants could be infringed upon.