I read David Freedman’s book Wrong a while ago. The article (which I have not read yet) appears to be a condensation of the book. I highlighted a few passages while I read. Here they are.
Quotes about the problem of groupthink from the chapter The Idiocy of Crowds:
…the notion that individuals tend to outthink, outdecide, and outperform groups is so well established that it isn’t even studied much anymore. “The average person certainly believes teamwork trumps individual work, but the evidence says otherwise,” says Natalie Allen, an organizational psychologist at the University of Western Ontario who has studied what she calls the “romance of teams.” …
Nearly four decades of research have exquisitely detailed the ineffectiveness of groups. The problems with collaborative and community thinking have been repeatedly highlighted by a stream of studies, starting with the Yale psychology researcher Irving Janis’s classic examination of “groupthink” back in 1972, which showed how groups could reach terrible decisions that none of the individuals in the group ever would have made on his own. As Janis and many others have shown—and as most of us know all too well—groups are frequently dominated not by people who are most likely to be right but rather by people who are belligerent, persuasive, persistent, manipulative, or forceful. Those who are even mildly adept at getting people to go along with them can quickly form small alliances of viewpoint that may in turn convince others to join in, eventually swaying even those with doubts—most of us don’t want to be the odd man out. …
Once a majority opinion is formed, even highly competent, confident people are reluctant to voice opinions that go against it…
…of the ten deadliest plane crashes in history, cockpit tapes reveal that six of them—killing a total of some 2,400 people—took place with at least one crew member being fully aware of the mistake that was about to bring the plane down but staying mostly quiet because the rest of the crew thought differently. …
Academic, financial, and clinical researchers submit to a pack mentality at least as easily as most sorts of groups or communities. “They go off together in the wrong direction, following one another like any collection of humans,” says Peter Sheridan Dodds, a University of Vermont mathematician who also does work in sociology and biology, among other fields. Herd thinking can keep the community trudging along in one direction for years, resistant to all kinds of contrary evidence …
There is almost no replication in science, and even when there is, the replication is ignored:
Okay, so lousy research can slip past peer review into journals. But surely as soon as other researchers put the published results to the test, the truth will out, right? Possibly—except that the vast majority of published research is never replicated or validated, or if it is, there is no record of it in research journals. All but the most prominent research tends to enter the records and forever persist as apparently legitimate by default. Martinson estimates that more than 95 percent of medical research findings aren’t replicated. No wonder: replication is more or less unfundable, and if someone does it on his own nickel, the results probably won’t come to light. Even a study that fails to replicate a published result, stated Nature in an editorial, “is unlikely ever to be published, or even submitted for publication.” In 2006 Nature reporter Jim Giles dug up the fact that two out of the four stories plastered on the cover of a 2002 issue of the journal—that is, half of the biggest stories in the world of science that week—had failed replication, without all that much notice being taken of it. …
Ioannidis, too, found evidence of the persistence of bad findings. He looked at studies reporting the cardiovascular benefits of vitamin E, anticancer benefits of beta-carotene, and anti-Alzheimer’s benefits of estrogen—important studies that were published in 1993, 1981, and 1996, respectively, and that were each convincingly and prominently refuted in one or more larger studies around 1999, 1994, and 2004, respectively. In 2005, the most recent year Ioannidis checked, half of the researchers who cited the original study of vitamin E did so in the context of accepting the original results, and through 2006 a little more than 60 percent cited the original beta-carotene and estrogen studies, though the results had been solidly refuted—thirteen years earlier in the case of beta-carotene.
Scientists have a substantial motive to fudge research or commit outright fraud:
What could motivate such surprisingly nontrivial apparent levels of dishonesty? The answer turns out to be pretty simple: researchers need to publish impressive findings to keep their careers alive, and some seem unable to come up with those findings via honest work. Bear in mind that researchers who don’t publish well-regarded work typically don’t get tenure and are forced out of their institutions. …
So researchers are pressured to come up with study results that are both interesting and positive. But Ioannidis, among many others, is quick to note a problem: the more surprising, novel, and exciting an idea, the less likely it is to be right.† An idea that seems highly likely to be true, that is utterly plausible, is probably not going to seem exciting—it’s the implausibility that often provides most of the novelty and enthusiasm. …
In other words, researchers are essentially highly incentivized to test exciting ideas that are likely to be wrong—and far more likely to be published. Back in 1989 economists at Harvard and the National Bureau of Economic Research estimated that virtually all published economic papers are wrong, attributing this astoundingly dismal assessment to the effects of publication bias. …
And therein lies the motivation to fudge research. If a study’s results don’t clearly support an interesting hypothesis, a researcher is free to stick it in a file drawer and possibly kiss his career or funding good-bye—or, alternatively, he can imagine salvaging the situation by fabricating data, or by doctoring the way the study is conducted in order to produce more attractive data, or by manipulating the analysis of the data, in order to come out with a dubious positive result.
Whistle-blowing is strongly discouraged:
Baltimore would later publicly apologize to O’Toole for the way she was treated, but she lost her job at MIT. The message sent to young researchers everywhere was clear: the science community isn’t a lot friendlier to whistle-blowers than are police departments and tobacco companies. There’s plenty of evidence that O’Toole’s experience accurately reflects the research culture. In a 1993 study, 53 percent of graduate students said they would expect reprisals for reporting a faculty member’s misconduct, and 26 percent of faculty members said they “probably or definitely” would expect retaliation for reporting a colleague—where retaliation can include blocking promotion or sending unflattering reviews of research to journals and funding committees.20 Only 18 percent of assistant professors stated they “definitely” could report conduct and not expect retaliation.
Publication bias, with the help of incompetence or by itself, can produce the same effect as fraud:
…most researchers can get highly publishable and utterly wrong findings simply as a matter of carelessness or oversight, perhaps goosed by a little bit of subtle gamesmanship, even if unconsciously. …
Now let’s imagine there’s interest among twenty teams of scientists in testing a novel scientific theory, bearing in mind that, as we’ve discussed, most novel theories are likely to be wrong. If they all end up with “significant” findings, then we can reasonably propose that nineteen of them will have failed to confirm the theory, quite correctly, and one team will suffer a data fluke that will have led it to mistakenly conclude its work has confirmed the theory. Guess whose study is most likely to be published?
Peer review has much less value in selecting valid science than is popularly believed:
After the Schön scandal, Nature quoted the Princeton professor, Nobel laureate, and former Bell Labs researcher Philip Anderson as saying, “Nature’s editorial and refereeing policy seems to be influenced by the newsworthiness of the work, not necessarily its quality. And Science seems to be caught up in a similar syndrome.” …
It’s peer review that, more than anything else, [in the popular imagination] is supposed to separate the genuine, reliable science served up in research journals from the apparently frequently junky stuff we get in the mass media. Unfortunately, anecdotal evidence that the peer-review process is an effective way of picking out lousy and even fraudulent work is not encouraging. A panel of researchers and editors assembled to advise Science magazine after the Hwang scandal issued a statement noting that reviewers don’t even look for fraud. …
In fact, it is typically science journalists and other outside observers who imagine peer review to be an assurance of study reliability, and less so scientists. “Scientists understand that peer review per se provides only a minimal assurance of quality, and that the public conception of peer review as a stamp of authentication is far from the truth,” Jennings has written.
By capturing the process of peer review, bad scientists can suppress refutation of their own results:
“Prestigious investigators may suppress via the peer-review process the appearance and dissemination of findings that refute their findings, thus condemning their field to perpetuate false dogma,” states Ioannidis.
In re the chilling effects of status: In Gawande’s The Checklist Manifesto, he says that one of the valuable effects of checklists is that they enable low status people (like nurses) to tell their “superiors” that some crucial step has been skipped.
I read David Freedman’s book Wrong a while ago. The article (which I have not read yet) appears to be a condensation of the book. I highlighted a few passages while I read. Here they are.
Quotes about the problem of groupthink from the chapter The Idiocy of Crowds:
There is almost no replication in science, and even when there is, the replication is ignored:
Scientists have a substantial motive to fudge research or commit outright fraud:
Whistle-blowing is strongly discouraged:
Publication bias, with the help of incompetence or by itself, can produce the same effect as fraud:
Peer review has much less value in selecting valid science than is popularly believed:
By capturing the process of peer review, bad scientists can suppress refutation of their own results:
In re the chilling effects of status: In Gawande’s The Checklist Manifesto, he says that one of the valuable effects of checklists is that they enable low status people (like nurses) to tell their “superiors” that some crucial step has been skipped.