Late last year, Nature made some noise with their experiment in open peer review. They invited authors, if they wished, to allow their submitted manuscripts to be opened for comments online. As the journal's summary of the process makes clear, though, very few were willing to try it. (My look at the situation back in September wasn't very encouraging, either).
Only 5% of the papers sent out for review during the trial were made available by their authors, and these 71 papers went on to attract very little notice on the peer-review site itself. The editors counted 92 comments total, despite decent amounts of traffic and numerous attempts to drum up interest. As an aside, the statistics would seem to indicate a power-law distribution, which should surprise no one, with a few papers getting a strong plurality of the comments and many getting none at all.
The experiment has ended, and Nature is through with the idea of open peer review for now. But you still have to wonder if there's merit there - and if so, what flaws led to this underwhelming performance. I'd like to suggest one gigantic, ground-shaking factor, which sat right down and made itself at home immediately: lack of anonymity. Does anyone doubt that one reason that traditional peer review works (to the extent that it does) is its anonymous format? Would people be willing to say the things that they do about papers that they're refereeing if they knew that the authors would have their e-mail address and phone number?
I sure wouldn't. In my career I've reviewed for several journals, and even though I've let more papers through than I've rejected, sometimes my assent has been less than enthusiastic - on the order of "Well, this isn't too exciting, but you publish lots of other things just like it, so why not?" And I'm pretty hard on abuses of literature citations - like not citing clearly anticipatory work from some other group, or results which challenge the paper's own conclusions. "Publish only if they fix this part" is a common report from me in those cases. When I do reject a paper, it's not because it's a borderline case, though, because I try to let those through (with suggestions for improvement).
I would feel very inhibited indeed if I knew that the authors of the papers I've criticized were able to contact me directly (or to file my name in their "To Have Dirt Done To" folder). Many more comments would have come in during Nature's experiment if people had the ability to use anonymous screen names (and had the assurance that their real identities would not be revealed, just as in standard peer review). The journal could have reserved to right to delete libelous comments, of course, which in Great Britain is a pretty wide mandate.
But how many authors would have willingly offered up their papers to such a process? Probably far fewer than the 5% who tried system as it was, I'd say. But think of how our current peer review system would go over if we didn't have it already. "You're telling me," says the distinguished first author, "that you're going to send my paper out and let various unknown people rip into it? And you'll never tell me who they are? And I don't get a say in any of it? And this is going to be a major factor in whether my paper even gets accepted?" Imagine dropping that on an unprepared scientific community. It'd never fly
(Note: John Timmer at Ars Technica had a similar response to the anonymity issue, as did Information Week. TechDirt delivers another obituary).