I agree with previous comments about publishing in journals being an important status issue, but I think there is other value as well which is being ignored. For all of its annoyances and flaws, one good thing about peer review is that it really makes your paper better. When you submit a pretty good paper to a journal and get back the “revise and resubmit” along with the detailed list of criticisms and suggestions, then by the time the paper actually makes it into the journal, chances are that it will have become a really good paper.
But to return to the issue of papers being taken more seriously when published in a journal, I think that this view is actually quite justified. For researchers who are not already very knowledgeable in the precise area that is the topic for a given paper, whether or not the paper has withstood peer review is a very useful heuristic cue toward how much weight you should place on it. Basically, peer review keeps the author honest. An author posting a paper on his website can say pretty much whatever he wants. One of the purposes of peer reviewers is to make sure that the author isn’t making unreasonable claims, mischaracterizing theoretical positions, “reviewing” the relevant previous literature in a grossly selective way, etc. Like I said, if someone is already very familiar with the area, then they can evaluate these aspects of the paper for themselves. But if you’d like to communicate your position to a wider academic audience, peer review can help carry your paper a longer way.
If you haven’t passed peer review, it’s almost certainly because you can’t rather than because you have better things to do. If it’s not published in a peer-reviewed journal, there’s no reason to treat it any differently than the ramblings of the Time Cube guy.”
Perelman’s proof of the Poincare conjecture was never published in an academic journal, but was merely posted on arXiv. If that’s not science, then being correct is more important than being “scientific”.
Perelman’s proof has been published, e.g. this by the AMS, which has a rigorous refereeing process for books, and this in Asian Journal of Math with a more controversial refereeing process.
Though Perelman’s preprints appeared in 2002 and 2003, the Clay prize (which Perelman turned down) was not offered to him until last year, because the rules stipulate that the solutions to the prize problems have to stand unchallenged in published, peer-reviewed form for a certain number of years.
I’m not really familiar with the topic matter here, but I want to note that Michael Nielsen contradicts what you said (though Nielsen isn’t exactly an unbiased source here as an Open Science advocate):
Perelman’s breakthrough solving the Poincare conjecture ONLY appeared at the arXiv
The important point is that it doesn’t appear that Perelman produced the paper for publishing in a journal, but he made it and left it on the arXiv, which was later (you claim) published in journals. That’s quite a different view than “if it’s not published, it’s not science”
Indeed. However, you’ve raised a single remarkable exception to a general heuristic as if a single example is all that is needed to thorougly refute a general heuristic, and of course that’s not the case.
The overwhelming majority of papers put on arXiv and nowhere else are:
[ ] comparable to Perelman’s proof of the Poincare conjecture
[ ] not comparable to Perelman’s proof of the Poincare conjecture?
It’s not clear to me what the disagreement is here. Which heuristic are you defending again?
If it’s not published, it’s not science
Response: Can we skip the pointless categorizations and evaluate whether material is valid or useful on a case by case basis? Clearly there is some material that has not been published that is useful (see: This website).
If it’s not published in a peer-reviewed journal, there’s no reason to treat it any differently than the ramblings of the Time Cube guy.
Response: Ahh yes, anything not peer-reviewed clearly contains Time Cube-levels of crazy.
Or none of the above? I’m not sure we actually disagree on anything here.
The problem of publication bias is another reason to be wary of the publication heuristic recommended a few comments above. If you follow that heuristic rigorously, you will necessarily expose yourself to the systematic distortions arising from publication bias.
This is not to say that you should therefore believe the first unpublished paper you come across. It’s only to point out that the publication heuristic has certain problems, and while not ignored, it should be supplemented. You ignore unpublished research at your peril. In an ideal world, peer review filters the good from the bad and nothing else. We do not live in an ideal world, so caveat lector.
The process of journal publication is also extremely slow, so that refusal to read unpublished research threatens to retard your progress. This link gives time to publication for several journals—the average appears to be well over a year and approaching two years. What’s two years in Internet Time? Pretty long.
Most of the math papers are not comparable to Perelman’s proof in importance( that should be obvious) but most of them are mathematically correct and interesting to mathematicians. People will often see something on the arXiv and look at it. On the other hand, as I mentioned, people are also more likely to look at a preprint if they know it is actually accepted in a reputable journal.
If you want to play that game, then it’s not clear to me that the SIAI is doing “science” either, given that the focus is on existential risk due to AI (more like “philosophy” than “science”) and formal friendliness (math).
I think a better interpretation of your quote is to replace the word “science” with “disseminated scholarly communication.”
Essentially yes, though he might have had to individually contact a few mathematicians to make his existence known. Consider the example of Ramanujan. Wpedia:
In 1912–1913, he sent samples of his theorems to three academics at the University of Cambridge. Only Hardy recognized the brilliance of his work, subsequently inviting Ramanujan to visit and work with him at Cambridge.
Do we know whether there are many Ramanujans who gave up before getting through to someone? One way to tell might be to look at such people who had already given up and were then discovered through a coincidence.
It would be nice to have additional data. However, I think we can mine the case of Ramanujan for clues about difficulty of entry. What I find striking is that he only contacted three mathematicians. Had he contacted, say, twenty before being noticed, that would have suggested a higher barrier to entry. But it was apparently just three. My own experience is that the great scientific minds are very approachable, aside from a tiny handful of scientist celebrities who understandably have to learn to be less approachable.
I agree with previous comments about publishing in journals being an important status issue, but I think there is other value as well which is being ignored. For all of its annoyances and flaws, one good thing about peer review is that it really makes your paper better. When you submit a pretty good paper to a journal and get back the “revise and resubmit” along with the detailed list of criticisms and suggestions, then by the time the paper actually makes it into the journal, chances are that it will have become a really good paper.
But to return to the issue of papers being taken more seriously when published in a journal, I think that this view is actually quite justified. For researchers who are not already very knowledgeable in the precise area that is the topic for a given paper, whether or not the paper has withstood peer review is a very useful heuristic cue toward how much weight you should place on it. Basically, peer review keeps the author honest. An author posting a paper on his website can say pretty much whatever he wants. One of the purposes of peer reviewers is to make sure that the author isn’t making unreasonable claims, mischaracterizing theoretical positions, “reviewing” the relevant previous literature in a grossly selective way, etc. Like I said, if someone is already very familiar with the area, then they can evaluate these aspects of the paper for themselves. But if you’d like to communicate your position to a wider academic audience, peer review can help carry your paper a longer way.
-- Paraphrase of a speaker at the Northeast Conference on Science and Skepticism
If it’s not published, it might be correct, but it’s not science.
Perelman’s proof of the Poincare conjecture was never published in an academic journal, but was merely posted on arXiv. If that’s not science, then being correct is more important than being “scientific”.
Perelman’s proof has been published, e.g. this by the AMS, which has a rigorous refereeing process for books, and this in Asian Journal of Math with a more controversial refereeing process.
Though Perelman’s preprints appeared in 2002 and 2003, the Clay prize (which Perelman turned down) was not offered to him until last year, because the rules stipulate that the solutions to the prize problems have to stand unchallenged in published, peer-reviewed form for a certain number of years.
I’m not really familiar with the topic matter here, but I want to note that Michael Nielsen contradicts what you said (though Nielsen isn’t exactly an unbiased source here as an Open Science advocate):
The important point is that it doesn’t appear that Perelman produced the paper for publishing in a journal, but he made it and left it on the arXiv, which was later (you claim) published in journals. That’s quite a different view than “if it’s not published, it’s not science”
Indeed. However, you’ve raised a single remarkable exception to a general heuristic as if a single example is all that is needed to thorougly refute a general heuristic, and of course that’s not the case.
The overwhelming majority of papers put on arXiv and nowhere else are:
[ ] comparable to Perelman’s proof of the Poincare conjecture
[ ] not comparable to Perelman’s proof of the Poincare conjecture?
It’s not clear to me what the disagreement is here. Which heuristic are you defending again?
Response: Can we skip the pointless categorizations and evaluate whether material is valid or useful on a case by case basis? Clearly there is some material that has not been published that is useful (see: This website).
Response: Ahh yes, anything not peer-reviewed clearly contains Time Cube-levels of crazy.
Or none of the above? I’m not sure we actually disagree on anything here.
The problem of publication bias is another reason to be wary of the publication heuristic recommended a few comments above. If you follow that heuristic rigorously, you will necessarily expose yourself to the systematic distortions arising from publication bias.
This is not to say that you should therefore believe the first unpublished paper you come across. It’s only to point out that the publication heuristic has certain problems, and while not ignored, it should be supplemented. You ignore unpublished research at your peril. In an ideal world, peer review filters the good from the bad and nothing else. We do not live in an ideal world, so caveat lector.
The process of journal publication is also extremely slow, so that refusal to read unpublished research threatens to retard your progress. This link gives time to publication for several journals—the average appears to be well over a year and approaching two years. What’s two years in Internet Time? Pretty long.
Let’s reword that: the overwhelming majority of papers published in peer-reviewed journals are comparable/not comparable to Perelman’s proof?
Probable answer: not remotely comparable. In fact, a lot of them are just plain wrong.
Most of the math papers are not comparable to Perelman’s proof in importance( that should be obvious) but most of them are mathematically correct and interesting to mathematicians. People will often see something on the arXiv and look at it. On the other hand, as I mentioned, people are also more likely to look at a preprint if they know it is actually accepted in a reputable journal.
Indeed, math isn’t science. ;)
I wonder—if Perelman was just “some guy” with no reputation as a mathematician, would anyone have noticed when he uploaded his proof?
If you want to play that game, then it’s not clear to me that the SIAI is doing “science” either, given that the focus is on existential risk due to AI (more like “philosophy” than “science”) and formal friendliness (math).
I think a better interpretation of your quote is to replace the word “science” with “disseminated scholarly communication.”
Good point.
Essentially yes, though he might have had to individually contact a few mathematicians to make his existence known. Consider the example of Ramanujan. Wpedia:
Do we know whether there are many Ramanujans who gave up before getting through to someone? One way to tell might be to look at such people who had already given up and were then discovered through a coincidence.
It would be nice to have additional data. However, I think we can mine the case of Ramanujan for clues about difficulty of entry. What I find striking is that he only contacted three mathematicians. Had he contacted, say, twenty before being noticed, that would have suggested a higher barrier to entry. But it was apparently just three. My own experience is that the great scientific minds are very approachable, aside from a tiny handful of scientist celebrities who understandably have to learn to be less approachable.