This was a fascinating post, but I found a surprising statement in the introduction:
“who are shy about telling us when their peers’ work is completely wrong.”
This runs deeply against my experience. I would say writing a paper gleefully proving your peers wrong is second only to writing a paper with an important new discovery in terms of academic satisfaction. In the middle of one controversy a colleague claimed (or maybe quoted) “Every paper published is a shot fired in a war”.
This is obviously running counter to your experience and I wonder how you came to that conclusion? Are we talking about well-cited papers that are “completely wrong”—or just that newer papers have effectively replaced them in the corpus.
Good question. I’d say: writing a paper proving your peers wrong is great fun, but requires a paper. You are expected to make a strong, detailed case, even when the work is pretty obviously flawed. You can’t just ignore a bad model in a background section or have a one-sentence “X found Y, but they’re blatantly p-hacking”—those moves risk a reviewer complaining. And even after writing the prove-them-wrong paper, you still can’t just ignore the bad work in background sections of future papers without risking reviewers’ ire.
Comment on “paper x” to my mind is the usual vehicle for complaining about faulty methods and poor statistical analysis. Since journals that accept comments tend to give a right of reply, review can be pretty light.
I would agree though that commenting on flaws like this is not as satisfying (mostly) as proper paper where an alternative hypothesis is promoted and opponents flaws lightly commented on. It is still a lot of work to comment and not a lot of point unless driving new science other than ego-tripping.
However, my original point remains—I don’t think researchers are remotely shy about criticizing the work of their peers.
This was a fascinating post, but I found a surprising statement in the introduction:
“who are shy about telling us when their peers’ work is completely wrong.”
This runs deeply against my experience. I would say writing a paper gleefully proving your peers wrong is second only to writing a paper with an important new discovery in terms of academic satisfaction. In the middle of one controversy a colleague claimed (or maybe quoted) “Every paper published is a shot fired in a war”.
This is obviously running counter to your experience and I wonder how you came to that conclusion? Are we talking about well-cited papers that are “completely wrong”—or just that newer papers have effectively replaced them in the corpus.
Good question. I’d say: writing a paper proving your peers wrong is great fun, but requires a paper. You are expected to make a strong, detailed case, even when the work is pretty obviously flawed. You can’t just ignore a bad model in a background section or have a one-sentence “X found Y, but they’re blatantly p-hacking”—those moves risk a reviewer complaining. And even after writing the prove-them-wrong paper, you still can’t just ignore the bad work in background sections of future papers without risking reviewers’ ire.
Does that fit your experience?
Comment on “paper x” to my mind is the usual vehicle for complaining about faulty methods and poor statistical analysis. Since journals that accept comments tend to give a right of reply, review can be pretty light.
I would agree though that commenting on flaws like this is not as satisfying (mostly) as proper paper where an alternative hypothesis is promoted and opponents flaws lightly commented on. It is still a lot of work to comment and not a lot of point unless driving new science other than ego-tripping.
However, my original point remains—I don’t think researchers are remotely shy about criticizing the work of their peers.
“Every paper published is a shot fired in a war”
Epistemic virtue isn’t a good strategy in that war, I suspect. Voicing your true best guesses is disincentivized unless you can prove them.