Online discussion is better than pre-publication peer review

Re­lated: Why Aca­demic Papers Are A Ter­rible Dis­cus­sion Fo­rum, Four Lay­ers of In­tel­lec­tual Conversation

Dur­ing a re­cent dis­cus­sion about (in part) aca­demic peer re­view, some peo­ple defended peer re­view as nec­es­sary in academia, de­spite its flaws, for time man­age­ment. Without it, they said, re­searchers would be over­whelmed by “cranks and in­com­pe­tents and time-card-punch­ers” and “semi-se­ri­ous peo­ple post ideas that have already been ad­dressed or re­futed in pa­pers already”. I replied that on on­line dis­cus­sion fo­rums, “it doesn’t take a lot of effort to de­tect cranks and pre­vi­ously ad­dressed ideas”. I was prompted by Michael Arc and Stu­art Arm­strong to elab­o­rate. Here’s what I wrote in re­sponse:

My ex­pe­rience is with sys­tems like LW. If an ar­ti­cle is in my own spe­cialty then I can judge it eas­ily and make com­ments if it’s in­ter­est­ing, oth­er­wise I look at its votes and other peo­ple’s com­ments to figure out whether it’s some­thing I should pay more at­ten­tion to. One ad­van­tage over peer re­view is that each spe­cial­ist can see all the un­filtered work in their own field, and it only takes one per­son from all the spe­cial­ists in a field to rec­og­nize that a work may be promis­ing, then com­ment on it and draw oth­ers’ at­ten­tions. Another ad­van­tage is that no­body can make ill-con­sid­ered com­ments with­out suffer­ing per­sonal con­se­quences since ev­ery­thing is pub­lic. This seem like an ob­vi­ous im­prove­ment over stan­dard pre-pub­li­ca­tion peer re­view, for the pur­pose of fil­ter­ing out bad work and fo­cus­ing at­ten­tion on promis­ing work, and in prac­tice works rea­son­ably well on LW.

Ap­par­ently some peo­ple in academia have come to similar con­clu­sions about how peer re­view is cur­rently done and are try­ing to re­form it in var­i­ous ways, in­clud­ing switch­ing to post-pub­li­ca­tion peer re­view (which seems very similar to what we do on fo­rums like LW). How­ever it’s trou­bling (in a “civ­i­liza­tional in­ad­e­quacy” sense) that academia is mov­ing so slowly in that di­rec­tion, de­spite the nec­es­sary en­abling tech­nol­ogy hav­ing been in­vented a decade or more ago.