credibility.com
In the ideal world, all of human knowledge could be accessed and evaluated by every individual for personal decisions; we are more towards having more information be accessible, but it is increasingly infeasible for individuals to process all the information relevant to all their questions. The solution is to split some common important questions into sub-questions and to rely on the reports of individuals who investigate specific questions, often themselves relying on the reports of others in addition to primary data (observations). But one cannot trust these reports completely; thus there is a need for a system which can evaluate the reviews and reviewers themselves. Reputation and later, peer review, has historically played this role; but now the technology exists to implement something like a “credibility.com″ in which every information source can be reviewed. Could such a site, properly implemented, grow to supersede the role now played by peer review?
There are issues with practicality—there simply isn’t time to review everything that is published. I recall that Wikipedia toyed with this idea (they wanted to have admins approve all edits) but there weren’t enough people willing to act as peer reviewers to get the job done. Also, this would not be a very prestigious (or interesting) position for a scientist, and so it would not be an in-demand job unless it were highly incentivized. If we believe David Hull’s explanation of incentives in science, which claims that scientists seek recognition by being cited, then being a reviewer is anathema.
See Take On It.
Peer review isn’t about judging the credibility of people. It’s about judging the credibility of articles. In modern times it’s also about telling the authors what they can do to improve their manuscripts.
Just because sometimes is right one time doesn’t mean that he should be trusted in a different case.