Rethinking Peer Review As The World Peer Reviews Claimed Proof That P≠NP

We recently discussed how incredibly broken the traditional scientific journal system is, in terms of how they tend to lock up access to information. However, that left out a much bigger concern: the peer review system they use doesn’t always work very well. There is, of course, the famous case of Hendrik Schön, who was the toast of the physics world, until it was discovered that his “breakthroughs” were frauds — even though they were peer reviewed. But that, of course, is an extreme case. Even outside of that, though, peer review has always been somewhat questionable, and many have warned in the past that it’s not particularly reliable or consistent in judging the quality of research.

This week, the world has been taken by storm by claims from Vinay Deolalikar, that he hasproved P≠NP, one of (if not the) biggest problem in math and computer science which haspotentially huge implications (pdf). However, what’s interesting is that the paper started getting a ton of attention prior to any sort of peer review… but all of the attention around it has resulted in people (experts and non-experts alike) around the world beginning to take part in a self-organizing peer review on the fly.

This is leading some to point out that this seems to be a much better method of peer reviewand should be looked at more seriously (found via Glyn Moody). Apparently, people are realizing that a much more open post-publication peer review process, where anyone can take part, is a lot more effective:

We are starting to see examples of post-publication peer review and see it radically out-perform traditional pre-publication peer review. The rapid demolition [1, 2, 3] of the JACS hydride oxidation paper last year (not least pointing out that the result wasn’t even novel) demonstrated the chemical blogosphere was more effective than peer review of one of the premiere chemistry journals. More recently 23andMe issued a detailed, and at least from an outside perspective devastating, peer review (with an attempt at replication!) of a widely reportedScience paper describing the identification of genes associated with longevity. This followed detailed critiques from a number of online writers.

The post goes on to discuss some of the pros and cons of this kind of post-publication peer review, and to respond to some of the claimed challenges to it as well. Obviously, this particular paper is a bit different in that the fame around the problem, and the interest in the story itself has a lot more people willing to just go and dig into the proof, but that doesn’t mean there aren’t some useful lessons to be learned here about getting past the old and increasingly ineffective method of peer review used in many places today.