[LINK] The Selected Papers Network
John Baez has been writing, here and here, about problems with the academic journal system and a tool that might be a step towards fixing them:
Last time Christopher Lee and I described some problems with scholarly publishing. The big problems are expensive journals and ineffective peer review. But we argued that solving these problems require new methods of
• selection—assessing papers
and
• endorsement—making the quality of papers known, thus giving scholars the prestige they need to get jobs and promotions.
The Selected Papers Network is an infrastructure for doing both these jobs in an open, distributed way. It’s not yet the solution to the big visible problems—just a framework upon which we can build those solutions. It’s just getting started, and it can use your help.
I should post separately about this at some point.
Suppose we have a Collective Judgment of Science system in which scientific karma enters the system at highly agreed-upon points, e.g. very well-replicated, significant findings. Is there a system with the following properties:
The karma entry points need not necessarily be the most trusted people. Let’s say you made a significant discovery, but 70% of the field disagrees with most of your opinions, and someone who hasn’t made a significant discovery is trusted by 95% of the people who make significant discoveries. We should perhaps believe the latter person over you; making one discovery is not proof of perfect epistemic reliability.
If someone goes rogue and endorses a thousand trolls, who in turn endorse a million trolls, the million trolls can do no more karmic damage / produce no more karmic distortion, than the original person.
If I make three significant discoveries or write three good papers, there is no incentive to spread those papers out over 3 pseudonyms, or coauthor them with 3 others, in terms of how much influence I will have afterward. There may potentially be some incentive to centralize, although this would also not be good.
Downvoting or strongly downvoting an idea that many reliable epistemic voters think is correct may potentially be taken as Bayesian evidence by the system that you sometimes downvote good ideas. It’s probably worth distinguishing this from concluding that you sometimes upvote bad ideas, without separate evidence.
Rather than give people an incentive to waste labor by systematically downvoting everything that person X said, there is a centralized “I think this person is a complete idiot” button. After pressing this button, further systematic downvoting has no effect. Obviously the order of operations should not be significant here, i.e., this button must have as much effect as downvoting everything. Perhaps you might be asked to look at the person’s 3 highest-karma nodes and asked if you really want to downvote those too (vs. an “I hate most but not all things you say” rating) given that indicating “I uniformly hate everything you say” may then potentially reflect poorly on your reliability.
Within these constraints, it should be generally true that one person who’s gotten a large karma prize cannot outvote 100 people who were all endorsed by trusted epistemics with karma originating from sources outweighing that single prize.
We’re okay with this system using terabytes or even petabytes of memory to scale, so long as it’s not exabytes and it can compute updates in real time, or at least less than an hour.
Being able to run on upvotes and downvotes is great, failing that having people click on a 5-star level or a linear spectrum is about as much info as we should ask, since most users will not provide more info than this on most occasions. We could potentially have a standard 5-star scale which by leaving the mouse present for 5 seconds can go to 6 stars, or a 7-star rating which can be given once per month, or something. We can’t ask users to rate along 3 separate dimensions.
We should take into account that some people have pickier standards and downvote more easily or upvote more rarely than others, or conversely someone who endorses almost everything is only providing discriminatory Bayesian evidence about a threshold on the low end of the quality scale.
We can suppose that nodes are clustered in a 3-level hierarchy by broadest area, subject, and subspecialization but probably shouldn’t suppose any more clustering in the data than this. It’s possible we shouldn’t try to assess it at all.
A consequence of this system is that as a philosopher, you can potentially achieve great endorsement of your perspicacity, but only by convincing people who were upvoted by people who delivered well-replicated significant experimental results. This strikes me as a feature, not a bug. I don’t know of any particularly better way to decide which philosophers are reliable.
It can potentially be possible to bet karma on predictions subject to definite settlement a la a prediction market, since this can only operate to increase reliability of the system. If an open question that people opinionated about is definitely settled, anyone who was bold in predicting a minority correct answer should have their karma in some way benefit. Again we do not want an incentive to create pseudonyms to get independent karma awards here. (We can perhaps imagine such a question-node as a single source which endorses everyone who endorsed its correct answer.)
Presentation ordering of new nodes takes into account a value-of-information calculation, not just the highest confidence in current karma. (Obviously, under such a calculation, more prolifically voting users will see more recent nodes. This is also fine.)
Not that I know of, but Advogato’s trust metric limits the damage by a rogue endorser of many trolls with a calculation using maximum network flow. It doesn’t allow for downvotes.
If you allow downvoting and blocking all of someone’s nodes, that could be an incentive for the person to partition their publications into three pseudonyms, so that once the first is blocked, the others are still available.
Is it possible to get enough people interested in this to do something with it (like a website?)
It seems like it would take a herculean effort to get enough scientists interested and willing to participate. But then again, there may be many more scientists disillusioned with the academic journal system than I think.
I’d be happy to pioneer it on LW if it was a simple enough algorithm. StackOverflow, MathOverflow, Quora, possibly Reddit might be quite interested if it worked. (I don’t know if there’s acknowledged borrowing—keeping in mind that we borrowed all of Reddit’s code in the first place and it was under an open-source give-back license—but Reddit seems to have adopted LW’s highlight-recent-comments innovation, so there’s been some backflow.) Wikipedia is disintegrating under the weight of deletionists and would probably have to be rebooted more than healed, but Earth needs a Wikipedia. There are plenty of likely adopters / testers / provers in advance of the general scientific community if a superior karma algorithm can be found.
At least in mathematics my impression is that there are a lot. The Cost of Knowledge boycott against Elsevier has about 13,000 signatures at the moment. Discussion of this kind of issue in the mathematical community has been happening for awhile now, most prominently at places like Tim Gowers’ and Terence Tao’s blogs, but also on Google Plus.
I have found a new mathematical proof about the irrationality of sqrt(2). I have put it here:
http://protokol2020.wordpress.com/2013/06/01/had-there-been-two-naturals-a-and-d/
Now I am watching how the system will swallow it. Accept it or reject it. Or ignore it.
FYI, I have definitely seen that proof before, although I can’t recall where.
Try to Google it out, if you can!