How to offer peer review comments
I was recently asked by an editor to offer (single-blind) peer review comments for some upcoming academic work on the Singularity. Having never done this before, I sought out some literature on how to do it.
In case others find themselves in this position, here are some helpful papers I found:
Benos et al., How to review a paper
Guilford, Teaching peer review and the process of scientific writing
Provenzale & Stanley, A systematic guide to reviewing a manuscript
But I know that many Less Wrongers have been engaged in the peer review process for many years. If so, please offer your own comments or link to other helpful instructions!
Considering your knowledge about the topic, the usual level of care and effort you put into your writing, and your apparent enthusiasm for the task, I am pretty sure that you’ll do a better job than at least 99% of reviewers in practice.
In my experience from inside the academic sausage factory, most peer reviews are quite awful. (Bryan Caplan is one academic who openly agrees.) This is not very surprising considering that the work is difficult, tedious, unpaid, uncredited, without any penalties for doing a bad job, and usually passed on to otherwise overworked, underpaid, and apathetic grad students. If you just manage to understand the paper and offer any sensible criticism and suggestions, that will be magnificent by the usual standards.
With this in mind, these papers you cite strike me as rather detached from how things really work in practice. Such high and elaborate intellectual standards might be realistic for reviewers writing publicly and with some incentive to do a good job, but definitely not for the way anonymous peer review works presently.
There is a huge variance in the type of comments reviewers give. Keep in mind that because of the large number of submissions the Singularity book has received the editors will, I’m guessing, be rejecting most abstracts and so if you think the abstract should be accepted you need to provide compelling reasons to justify your decision.
One of the best ways to help an author would be to give him a full citation or link to writings that strongly relate to his topic but which he appears unaware of.
Reviewing an abstract is very different from reviewing a full paper so if you’re doing the former don’t necessarily rely on the advice of people explaining how you should do the latter.
There was a very interesting discussion of exactly this question (at least as it relates to the mathematics community) on mathoverflow recently.