Corante

About this Author
DBL%20Hendrix%20small.png College chemistry, 1983

Derek Lowe The 2002 Model

Dbl%20new%20portrait%20B%26W.png After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases. To contact Derek email him directly: derekb.lowe@gmail.com Twitter: Dereklowe

Chemistry and Drug Data: Drugbank
Emolecules
ChemSpider
Chempedia Lab
Synthetic Pages
Organic Chemistry Portal
PubChem
Not Voodoo
DailyMed
Druglib
Clinicaltrials.gov

Chemistry and Pharma Blogs:
Org Prep Daily
The Haystack
Kilomentor
A New Merck, Reviewed
Liberal Arts Chemistry
Electron Pusher
All Things Metathesis
C&E News Blogs
Chemiotics II
Chemical Space
Noel O'Blog
In Vivo Blog
Terra Sigilatta
BBSRC/Douglas Kell
ChemBark
Realizations in Biostatistics
Chemjobber
Pharmalot
ChemSpider Blog
Pharmagossip
Med-Chemist
Organic Chem - Education & Industry
Pharma Strategy Blog
No Name No Slogan
Practical Fragments
SimBioSys
The Curious Wavefunction
Natural Product Man
Fragment Literature
Chemistry World Blog
Synthetic Nature
Chemistry Blog
Synthesizing Ideas
Business|Bytes|Genes|Molecules
Eye on FDA
Chemical Forums
Depth-First
Symyx Blog
Sceptical Chymist
Lamentations on Chemistry
Computational Organic Chemistry
Mining Drugs
Henry Rzepa


Science Blogs and News:
Bad Science
The Loom
Uncertain Principles
Fierce Biotech
Blogs for Industry
Omics! Omics!
Young Female Scientist
Notional Slurry
Nobel Intent
SciTech Daily
Science Blog
FuturePundit
Aetiology
Gene Expression (I)
Gene Expression (II)
Sciencebase
Pharyngula
Adventures in Ethics and Science
Transterrestrial Musings
Slashdot Science
Cosmic Variance
Biology News Net


Medical Blogs
DB's Medical Rants
Science-Based Medicine
GruntDoc
Respectful Insolence
Diabetes Mine


Economics and Business
Marginal Revolution
The Volokh Conspiracy
Knowledge Problem


Politics / Current Events
Virginia Postrel
Instapundit
Belmont Club
Mickey Kaus


Belles Lettres
Uncouth Reflections
Arts and Letters Daily
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline

In the Pipeline

« You'll Be Safe Under Here. Maybe. | Main | Good Stuff and Bad Stuff »

January 4, 2007

Take Your Shots (For Real, This Time?)

Email This Entry

Posted by Derek

Late last year, Nature made some noise with their experiment in open peer review. They invited authors, if they wished, to allow their submitted manuscripts to be opened for comments online. As the journal's summary of the process makes clear, though, very few were willing to try it. (My look at the situation back in September wasn't very encouraging, either).

Only 5% of the papers sent out for review during the trial were made available by their authors, and these 71 papers went on to attract very little notice on the peer-review site itself. The editors counted 92 comments total, despite decent amounts of traffic and numerous attempts to drum up interest. As an aside, the statistics would seem to indicate a power-law distribution, which should surprise no one, with a few papers getting a strong plurality of the comments and many getting none at all.

The experiment has ended, and Nature is through with the idea of open peer review for now. But you still have to wonder if there's merit there - and if so, what flaws led to this underwhelming performance. I'd like to suggest one gigantic, ground-shaking factor, which sat right down and made itself at home immediately: lack of anonymity. Does anyone doubt that one reason that traditional peer review works (to the extent that it does) is its anonymous format? Would people be willing to say the things that they do about papers that they're refereeing if they knew that the authors would have their e-mail address and phone number?

I sure wouldn't. In my career I've reviewed for several journals, and even though I've let more papers through than I've rejected, sometimes my assent has been less than enthusiastic - on the order of "Well, this isn't too exciting, but you publish lots of other things just like it, so why not?" And I'm pretty hard on abuses of literature citations - like not citing clearly anticipatory work from some other group, or results which challenge the paper's own conclusions. "Publish only if they fix this part" is a common report from me in those cases. When I do reject a paper, it's not because it's a borderline case, though, because I try to let those through (with suggestions for improvement).

I would feel very inhibited indeed if I knew that the authors of the papers I've criticized were able to contact me directly (or to file my name in their "To Have Dirt Done To" folder). Many more comments would have come in during Nature's experiment if people had the ability to use anonymous screen names (and had the assurance that their real identities would not be revealed, just as in standard peer review). The journal could have reserved to right to delete libelous comments, of course, which in Great Britain is a pretty wide mandate.

But how many authors would have willingly offered up their papers to such a process? Probably far fewer than the 5% who tried system as it was, I'd say. But think of how our current peer review system would go over if we didn't have it already. "You're telling me," says the distinguished first author, "that you're going to send my paper out and let various unknown people rip into it? And you'll never tell me who they are? And I don't get a say in any of it? And this is going to be a major factor in whether my paper even gets accepted?" Imagine dropping that on an unprepared scientific community. It'd never fly

(Note: John Timmer at Ars Technica had a similar response to the anonymity issue, as did Information Week. TechDirt delivers another obituary).

Comments (15) + TrackBacks (0) | Category: The Scientific Literature


COMMENTS

1. KinasePro on January 5, 2007 12:05 AM writes...

"Well, this isn't too exciting, but you publish lots of other things just like it, so why not?"

I quite literally laughed out loud, as I have BMCL open in another tab...

Permalink to Comment

2. Derek Lowe on January 5, 2007 7:02 AM writes...

Hah! Now, did I say that that was a Bioorg. Med. Chem. Letters review?

Permalink to Comment

3. Colst on January 5, 2007 10:32 AM writes...

I would imagine you're right about the effects of a lack of anonymity.

Personally, I didn't comment because there was nothing about which I had enough knowledge to say anything useful. There wasn't a single chemistry paper. I think there needs to be a certain number of papers to make this sort of effort semi-successful. That Nature covers such a wide range of disciplines raises this number (that is, you need a number of papers in each subject). Even the, I imagine you'd have the power-law-like distribution.

In your September post about this, you said:
"What I'd like to see is an idea that's been proposed before, but never implemented in chemistry: comments on papers after they've published."
Nature says it's next peer-review experiment is in that direction.

Permalink to Comment

4. d_orbital on January 5, 2007 11:51 AM writes...

I wonder if something like this could work: Two (or so) reviewers are assigned to a manuscript that is also viewable on the web. This manuscript is open to viewer comments, who must be registered, except that those comments are not viewable by the public but go directly to the editor or reviewers. This way the papers are openly (but not publicly) criticized with a wide range of opinions at the reviewers (or editors) disposal. Thus you'd have two formal reviewers and a variety of other opinions (from legitimate researchers in the field) that can be sifted through by the editor (who now truly is a referee). Accountability would keep crackpots from saying whatever they want and anonymity would keep discussion open.

Permalink to Comment

5. Srikanth on January 5, 2007 12:30 PM writes...

Chemistry is very competitive and there are bad professors or competitors who want to make hard for small people like me. Every area has different standards for submission (characterization or number of positive results needed to make good story.) How anyone know what the real rules for a journal are?

SRIKANTH HAS A SIMPLE SOULTION TO REVIEW PROBLEM THAT WILL CLEAN UP WHOLE FIELD!

Keep reviewer anonymous but PUBLISH ACTUAL REVIEWS!

Then everyone know if journal is fair and everyone know the rules. Bad reviewer wont be able to play games and do evil things.

Yes I have other people write my papers but I still need to know what to do so big brain can write it up.

Permalink to Comment

6. milkshake on January 5, 2007 1:52 PM writes...

Maybe the problem was the journal also. People who publish in Nature take themselves more seriously.

Personaly I wouldn't mind - some comments from established referees can be pretty dumb also, I imagine that median response from people participating in open anonymous reviw process won't be that much worse.

Permalink to Comment

7. Eric Johnson on January 5, 2007 4:04 PM writes...

Yes, I agree with your analysis re anonymity, Derek.

I also think web-based pre- and/or post-publication commentary will eventually establish itself (why shouldn't or wouldn't it? spam?) -- whether or not today's ways of peer review are ultimately altered or supplanted. Speaking for myself, I sometimes find "rapid response" online comments on the British Med J to be of deep interest. The internet is no less useful for discoursing on science than it is for selling used books, adulating rock bands, or any other communicative pursuit. Exploitation of the new medium is naturally slower for science, simply because power rests with a bunch of really busy people aged over 50, and because traditional centers of life-giving cachet (such as the main pages of Nature and Science) clearly aren't going to be history any time soon.

Permalink to Comment

8. Eric Johnson on January 5, 2007 5:50 PM writes...

Here's something of topical interest:

http://www.nature.com/nature/journal/v445/n7123/full/445009a.html

Open-access journal will publish first, judge later
Jim Giles

PLoS One aims to challenge academia's obsession with journal status.

A radical project from the Public Library of Science (PLoS), the most prominent publisher in the open-access movement, is setting out to challenge academia's obsession with journal status and impact factors.

The online-only PLoS One, which launched on 20 December, will publish any paper that is methodologically sound. Supporters say the approach will remove some of the inefficiencies associated with current peer-review systems — but critics question whether a journal that eschews impact factors will manage to attract papers.

[...]

Every paper submitted to the journal is reviewed by at least one member of PLoS One's editorial board of over 200 researchers, but only to check for serious flaws in the way the experiment was conducted and analysed. In contrast to almost all other journals, referees ignore the significance of the result. Notable papers will instead be highlighted by the attention they attract after publication.

Visitors to the PLoS One website can, for example, attach comments to specific parts of a paper and rate the paper as a whole. Data from those systems, as well as download and citation statistics, will then allow PLoS One's editors to identify and promote the papers that researchers are talking about. "We're trying to make a journal where papers are not the end point, they are the start of a discussion," says PLoS One managing editor Chris Surridge, based in Cambridge, UK.

[...]

Permalink to Comment

9. Milo on January 5, 2007 6:11 PM writes...

Eric,

What is funny about that article is that those of us who do not subscribe cannot access it from home.

Permalink to Comment

10. Derek Lowe on January 5, 2007 6:22 PM writes...

PLoS One is going to be the subject of either Monday or Tuesday's post here, as a matter of fact. . .

Permalink to Comment

11. Brooks Moses on January 6, 2007 2:24 PM writes...

"But think of how our current peer review system would go over if we didn't have it already. "You're telling me," says the distinguished first author, "that you're going to send my paper out and let various unknown people rip into it? And you'll never tell me who they are? And I don't get a say in any of it? And this is going to be a major factor in whether my paper even gets accepted?" Imagine dropping that on an unprepared scientific community. It'd never fly."

There was an interesting article in Physics Today about this, six months ago or so, when they were opening up some of their older archives of peer-review communications to the APS, and came across Einstein's response to getting an unfavorable peer review on a paper. He was, indeed, very much not pleased -- to the point where he refused to submit to an APS journal for decades afterwards.

(Ironically, the unfavorable review turned out to be correct, and Einstein's conclusion in that paper to have been embarassingly wrong.)

Permalink to Comment

12. Scientific Debate on January 6, 2007 6:09 PM writes...

Some of the accompanying articles in Nature's online section on peer review mentioned the possibility of post-publication review of papers. In fact, we have just launched such an experiment, called Scientific Debate, which may be accessed at the url above.

It's focused on biological sciences at the moment, but if there is enough interest from the chemistry community, we would be happy to add chemistry categories. I point the site out here because I think that it's directly relevant to the discussion and to Colst's comments. I also hope that some of you might consider participating and that you might find the tool useful once the number of users reaches a critical mass.

Our objectives are (1) to serve as a post-publication filtering tool, to enable the highlighting of meritorious papers through a digg-style interface, and (2) to provide a forum for the frank discussion and evaluation of published papers. Unlike Faculty of 1000, we are free and we allow anyone to submit or comment on a paper. A comment rating system is built into the interface, and we really hope that this will promote a high level of intellectual discussion.

Keep in mind that we literally just launched. There are a few submissions, but if things seem a little rough around the edges, that's because they are, and we promise that we're working to improve things.

Any feedback would be greatly appreciated. Cheers!

Permalink to Comment

13. Deepak on January 8, 2007 12:13 PM writes...

Can't wait for your PLoS One article. I am becoming increasingly bugged with the scientific community. Are we so uncertain of our opinions that we refuse to defend them in public. When it suits us we don't shirk fame and glory, so why do we hide behind anonymity when it comes to public debate.

That said, if its such a big deal, we should all just have scientific avatars. Then our avatars can debate away.

Permalink to Comment

14. TFox on January 8, 2007 7:29 PM writes...

Biology Direct has another interesting model. Peer reviewed, but nonanonymously, and the reviews get published with the paper. You're supposed to find your own reviewers and get it reviewed before you submit. Obviously, this isn't designed as a screening mechanism.

Permalink to Comment

15. Eric J Johnson on January 8, 2007 8:47 PM writes...

Deepak, I quite agree with your perspective. In my areas (immunologic and infectious diseases, and antimicrobials), www.alzforum.org is the only rather active public forum I know of where the terminal degree holders regularly come out to play. (I haven't been there much though, not having focused on Alzheimer's much.)

When you think about it, perhaps the net actually offers better public dialogical possibilities than conferences and seminars do. How many talks have you seen where the questions and answers after the talk were really that great? Sometimes some of them are great, but at other times it seems to me that both sides of the dialogue are inhibited by both politeness and the attempt to look smart. Both sides try - on the fly - not to say something mean or dumb. In online communications, you can take your time - and you can look something up before you make your rejoinder. Language barriers are also less of a problem.

Of course, people do talk in the halls at conferences (and by email), and there is less performance pressure there than in the conference room; that's not public though.

Permalink to Comment

POST A COMMENT




Remember Me?



EMAIL THIS ENTRY TO A FRIEND

Email this entry to:

Your email address:

Message (optional):




RELATED ENTRIES
Scripps Update
What If Drug Patents Were Written Like Software Patents?
Stem Cells: The Center of "Right to Try"
Speaking of Polyphenols. . .
Dark Biology And Small Molecules
How Polyphenols Work, Perhaps?
More On Automated Medicinal Chemistry
Scripps Merging With USC?