We’ve also discovered the usefulness of peer review
I object, for reasons wonderfully stated by gwern here
Why do we need the process of peer review?
Peer review is not robust against even low levels of collusion (http://arxiv.org/abs/1008.4324v1). Scientists who win the Nobel Prize find their other work suddenly being heavily cited (http://www.nature.com/news/2011/110506/full/news.2011.270.ht...), suggesting either that the community either badly failed in recognizing the work’s true value or that they are now sucking up & attempting to look better by the halo effect. (A mathematician once told me that often, to boost a paper’s acceptance chance, they would add citations to papers by the journal’s editors—a practice that will surprise none familiar with Goodhart’s law and the use of citations in tenure & grants.)
Physicist Michael Nielsen points out (http://michaelnielsen.org/blog/three-myths-about-scientific-...) that peer review is historically rare (just one of Einstein’s 300 papers was peer reviewed! the famous Nature did not institute peer review until 1967), has been poorly studied (http://jama.ama-assn.org/cgi/content/abstract/287/21/2784) & not shown to be effective, is nationally biased (http://jama.ama-assn.org/cgi/content/full/295/14/1675), erroneously rejects many historic discoveries (one study lists “34 Nobel Laureates whose awarded work was rejected by peer review” (http://www.canonicalscience.org/publications/canonicalscienc...); Horribin 1990 (http://jama.ama-assn.org/content/263/10/1438.abstract) lists others like the discovery of quarks), and catches only a small fraction (http://jama.ama-assn.org/cgi/content/abstract/280/3/237) of errors. And fraud, like the one we just saw in psychology? Forget about it (http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjourna...);
“A pooled weighted average of 1.97% (N = 7, 95%CI: 0.86–4.45) of scientists admitted to have fabricated, falsified or modified data or results at least once –a serious form of misconduct by any standard– and up to 33.7% admitted other questionable research practices. In surveys asking about the behaviour of colleagues, admission rates were 14.12% (N = 12, 95% CI: 9.91–19.72) for falsification, and up to 72% for other questionable research practices....When these factors were controlled for, misconduct was reported more frequently by medical/pharmacological researchers than others.”
No, peer review is not the secret sauce of science. Replication is more like it. (Emphasis not in original)
That was actually just a slightly-edited-for-Hacker-News excerpt from my standing mini-essay explaining why we can’t trust science too much; the whole thing currently lives at http://www.gwern.net/DNB%20FAQ#fn51
I object, for reasons wonderfully stated by gwern here
That was actually just a slightly-edited-for-Hacker-News excerpt from my standing mini-essay explaining why we can’t trust science too much; the whole thing currently lives at http://www.gwern.net/DNB%20FAQ#fn51
That link points to your Dual N-Back piece. I think you meant https://www.gwern.net/Replication#nhst-and-systematic-biases