Corante

About this Author
DBL%20Hendrix%20small.png College chemistry, 1983

Derek Lowe The 2002 Model

Dbl%20new%20portrait%20B%26W.png After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases. To contact Derek email him directly: derekb.lowe@gmail.com Twitter: Dereklowe

Chemistry and Drug Data: Drugbank
Emolecules
ChemSpider
Chempedia Lab
Synthetic Pages
Organic Chemistry Portal
PubChem
Not Voodoo
DailyMed
Druglib
Clinicaltrials.gov

Chemistry and Pharma Blogs:
Org Prep Daily
The Haystack
Kilomentor
A New Merck, Reviewed
Liberal Arts Chemistry
Electron Pusher
All Things Metathesis
C&E News Blogs
Chemiotics II
Chemical Space
Noel O'Blog
In Vivo Blog
Terra Sigilatta
BBSRC/Douglas Kell
ChemBark
Realizations in Biostatistics
Chemjobber
Pharmalot
ChemSpider Blog
Pharmagossip
Med-Chemist
Organic Chem - Education & Industry
Pharma Strategy Blog
No Name No Slogan
Practical Fragments
SimBioSys
The Curious Wavefunction
Natural Product Man
Fragment Literature
Chemistry World Blog
Synthetic Nature
Chemistry Blog
Synthesizing Ideas
Business|Bytes|Genes|Molecules
Eye on FDA
Chemical Forums
Depth-First
Symyx Blog
Sceptical Chymist
Lamentations on Chemistry
Computational Organic Chemistry
Mining Drugs
Henry Rzepa


Science Blogs and News:
Bad Science
The Loom
Uncertain Principles
Fierce Biotech
Blogs for Industry
Omics! Omics!
Young Female Scientist
Notional Slurry
Nobel Intent
SciTech Daily
Science Blog
FuturePundit
Aetiology
Gene Expression (I)
Gene Expression (II)
Sciencebase
Pharyngula
Adventures in Ethics and Science
Transterrestrial Musings
Slashdot Science
Cosmic Variance
Biology News Net


Medical Blogs
DB's Medical Rants
Science-Based Medicine
GruntDoc
Respectful Insolence
Diabetes Mine


Economics and Business
Marginal Revolution
The Volokh Conspiracy
Knowledge Problem


Politics / Current Events
Virginia Postrel
Instapundit
Belmont Club
Mickey Kaus


Belles Lettres
Uncouth Reflections
Arts and Letters Daily
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline

In the Pipeline

« More From the Fourth | Main | Drug Prices in the US: Not So High After All? »

July 6, 2010

Commenting On Scientific Papers: How Come No One Does It?

Email This Entry

Posted by Derek

Why doesn't anyone comment on scientific papers? Let's narrow that down: why doesn't anyone comment on them when they have a comments field attached to them on a scientific publisher's web site?

Nature has been wondering about this for a while. In a new item about "Web 2.0" tools in science, they mention:

But deposition in arXiv is about as far as the scientific openness of even astronomers goes. The discussion that ensues is private. As Nature's experiment in open peer review showed (http://go.nature.com/N67mFk), and as can be seen from the lack of commenting on papers in Nature and other journals that encourage it, researchers see little to be gained from open discourse before or after publication. Not only are they busy, as the above quotes attest, but there's no credit to be gained, and some risk if one makes an erroneous or critical statement in public. What is more, astronomers and biologists register active discouragement of blogging — a form of communication that in their eyes carries no stamp of reliability or prestige. That picture of resistance to interactive discussion of science on the Internet is further amplified in a new survey, If You Build It, Will They Come? How Researchers Perceive and Use Web 2.0, to be published later this month by the UK Research Information Network.

Contrast that with a comment left here the other day, where a reader suggested that a great business plan might be to start a web site where people could comment on scientific publications. How to reconcile these world views? I think that one word goes a long way: anonymity. The first reaction to that is often a mental picture of the comments pages to (say) YouTube videos, and a brief shudder. And it's true that comments sections that allow easy anonymous accounts can attract all sorts of nasty stuff, as newspapers found out when they opened up their sites.

But anonymity has a long, distinguished history in science. Peer review! People will speak frankly about a paper under review, as long as they know that their comments will remain untraceable by the manuscript's authors. What makes it acceptable is that the editors of the journal know all the names involved, which (usually) keeps things above the midnight-ninja-assassination level.

So what would be needed, I think, would be a site where real e-mail and contact information would be required. You would be free to post under your real name, if you wanted to, or to take on whatever nom de guerre you wished, with the total assurance this this would not be revealed. At the same time, sheer invective and libel would be tossed out immediately, with the line to be drawn at the discretion of the site's editors. Personally, I'd delete comments that said only "This paper sucked", but I'd leave in the ones that said "This paper sucked because of X, Y, and Z." And the comments that started that way but then went on to talk about how the authors sucked in similar fashion would be truncated with some sort of standard mark meaning "ad hominem deleted".

That would be a lot of work. And there's a good chance that it would never take off at all, given the amount of trust involved. Would I wish to run such a site? No way - I already have a full-time job, thanks. But I'd like to see someone try.

Comments (26) + TrackBacks (0) | Category: The Scientific Literature


COMMENTS

1. Brian on July 6, 2010 7:59 AM writes...

Anonymous commenting site devoted to neuroscience:

http://thirdreviewer.com/
via:
http://scienceblogs.com/drugmonkey/2010/06/not_getting_the_point_on_the_t.php

I think there may be real potential in this model to undermine a traditional publisher's lock on mindshare to the point of, eventually, accepting manuscripts for "publication"

Permalink to Comment

2. Elizabeth on July 6, 2010 9:09 AM writes...

There's also, at least for me, in the way in which I read papers than in how I read blogs. For blogs, I read at the site and am right there to comment. For papers, I usually pull several PDFs on a topic to read later... at which point I'd have to go back to the journal to leave my thoughts. It's not an excessively high barrier, but it is one.

Permalink to Comment

3. Verpa on July 6, 2010 9:40 AM writes...

As you probably know, there are several resources for commenting on chemistry papers ( two run by Mitch of the Chemistry-Blog.com are: chemfeeds.com and reddit.com/r/chemistry, there is also what I like to call the ACS ghetto: http://pubs.acs.org/JACSbeta/journalclub/ ) but none are likely to take off unless the publishers themselves make a serious attempt to build a community instead of wading out a few feet into the future.

I've never been able to decide if I think the whole 'I'll sound stupid' excuse was a canard. Two things that make me think it is:
1. Everyone sounds stupid occasionally at group meetings ... if you don't, I don't think you're speaking up enough, and that's in front of people reasonably versed in the same subject who you'll be seeing the next day.
2. Who takes things said on the internet quite that seriously...personally I wouldn't judge a person for a stupid comment online, unless it was a string of stupid comments. Adding a means to retract a comment in addition to your anonymity idea would probably work too.

I'm going on a bit here, but one last point, I think the real issue is the game-theory of it. The benefits accrued from a commenting system mainly go to the journal/site and the reviewed, not the commenter. In the current model, reviewers get the prestige of being reviewers. Until we can work out a method that accrues some benefit to the commenter, it won't probably happen.

Maybe profs when hiring grad students / post docs could ask to look at their comments on papers in the group's field ... would benefit the profs ( can see the level of thought in the student ) and the student ( would have another means of standing out when applying ). Ok, that'd be a terrible idea, but you can kind of see my game theory point.

Permalink to Comment

4. John Harrold on July 6, 2010 9:46 AM writes...

Elizabeth,

I think this is a technical hurdle that can be easily overcome. The PDF document structure supports hyperlinks, and the publisher could easily embed a link to the discussion somewhere on the first page. Then after you read, you can scroll to the top, click on the link, and start commenting. It would be difficult to get the publishers on board, but the solution is there.

Permalink to Comment

5. Student on July 6, 2010 9:48 AM writes...

We need more opinions in chemistry. It's not like someone's going to get up in the Q&A session after a talk and say, "Dude, that topic is boring as hell, why are you using up good talent and resources investigating that trivial problem when there are so many important problems out there?" The only questions that are usually asked at seminars are not "big picture" questions, but are more "trivial" and involve data analysis or interpretations of the data. The only time you get honest "big picture" feedback is in grant reviews and anonymous paper peer review, unless you have a really close friends who are comfortable saying such things to you. We need more mechanisms for anonymous, honest opinions.

This is because grad students are trained in how to collect data and analyze results, but get almost no training in how to have good scientific taste. Good taste is a subjective opinionated thing, and since we never express opinions very much, but rather focus on the facts, students get the impression that this aspect is unimportant. But it is REALLY important; indeed, having good taste may be the most important thing for an R1 professor. Consequently, many students leave grad school with no feel for how to select an important problem, or to have good taste in what new problem or field to start, because no one wants to openly express an opinion, which frequently involves criticizing some area of chemistry (which their colleagues might be engaged in) or suggesting that a topic or problem is less important than something else.

Permalink to Comment

6. Nick K on July 6, 2010 9:55 AM writes...

Perhaps chemistry journals could have a "Letters to the Editor" section like newspapers.

Permalink to Comment

7. Andrew on July 6, 2010 11:00 AM writes...

This isn't an uncommon problem in other arenas either. Any forum that is remotely open is likely to attract a lot of useless comments and discourage constructive dialog. Wikipedia is a great example of this, and they try really, really hard to develop just such an environment.

www.daringfireball.net is run by a prominent tech blogger, John Gruber, who does not allow comments on his site. Instead, he posts his own comments on his site and links to the stories or blogs or articles that are of interest to him and any dialog is handled by the respective blogs of the people he is discussing an issue with, even with people that he's not particularly a fan of.

In so doing, no crap collects on his site, and everyone else is free to post and manage their own online persona as they see fit, allowing comments, or not, as appropriate.

The reality is that the forum for all this is already there - the internet itself - just a matter of getting people to take more ownership of their own online presence. I would host my own blog and be far more comfortable with complete control over that content than have to join several different forums and maintain a presence in each with almost zero control just to be able to be part of those communities.

Prominent researchers, students, people who publish papers or other findings should be able to post or publish their findings in places where they can be easily seen, reviewed and discussed by their peers such as online journals and the like, as well as maintain their own blog about their area of expertise where they can post follow-up information and commentary and links to relevant supporting material or discussion on other sites or forums.

Permalink to Comment

8. Cloud on July 6, 2010 11:09 AM writes...

You could try crowdsourcing the moderation, like Slashdot does.

It isn't a perfect system, but it does OK. Most flamebait comments are quickly marked as such.

Permalink to Comment

9. Anonymous on July 6, 2010 11:59 AM writes...

In cancer research, no one comments on anybody else's papers and no one ever says anything critical in anything but a completely private setting between friends.

Most of the cancer research literature (including many/most papers in the most high profile journals) does not represent biological reality and is populated by the same inside group who reviews each others papers and who then write the News and Views type articles on the papers they accept (which they then put on their own cv as a Nature paper).

There is literally nothing to be gained from criticizing these papers, and alot to be lost if confidentiality was broken. If any of the people you criticized were the reviewers of your later grants or papers, you would be in big trouble. Sad, but true.

Permalink to Comment

10. mitch on July 6, 2010 11:59 AM writes...

Anyone can leave comments on chemistry journals at http://www.chemfeeds.com/

Permalink to Comment

11. Virgil on July 6, 2010 12:36 PM writes...

I don't see the journals being particularly interested in opening up for comments, for one chief reason...

If, via comments, it is revealed that there's a major flaw with the paper, the journal then has to pretty much admit that it was not reviewed properly in the first place.

There have been some spectacular examples of failure in peer review (e.g. the Warda & Han affair over at Proteomics a couple of years ago), which have never been satisfactorily resolved, to the standards of the masses on the InterNet.

One example from my own experience... at a Nature journal, reading a published paper and finding some major issues with it (results that contradicted all that was known in the field at the time, plus huge problems with labeling of figures). We wrote a letter to the editor (including novel unpublished data of our own to bolster our case) - it was rejected because we had "failed to prove, using the same methods, that anything in the paper was wrong". We wrote back, this time with more data (multiple tissues, multiple strains, from multiple labs), and 15 authors including some very heavy hitters in the field.

They sent our 2 page letter to 6 reviewers. After a year of back-and-forth, they eventually published our letter, greatly edited. In the proofs stage, they demanded we remove some of our comments, because this was dealt with in an erratum to accompany the author's response to our letter. We refused - without this, there would be nothing for the original authors to respond to! Bottom line - the journal knew that having these bad things in the paper exposed was tantamount to admitting it was poorly reviewed in the first place.

Throughout the entire process, the editorial office of the journal was a complete bear. Anyone would think we had questioned their parents' marital status at time of birth. They just simply could not bring themselves to admit that they had screwed up the peer review process. Given such craziness, I think it will be a long time before journals are happy to open up their papers for true free-for-all on-line commenting.

Permalink to Comment

12. Anon 2 on July 6, 2010 4:43 PM writes...

I don't see the point. Sometimes, one actually should get away from a compouter & do some of your own work, set up your own experiments and interpret the results instead of simply reading a web site and being a "wise skeptic"

Increasingly, I see people preferring to comment on other peoples results rather than getting-on with the need to generate new hypotheses, and go through the often tedious process of conducting experiments to test them. In my company, there seem to be an increasingly large number of people who talk about data, and fewer who generate it. And while this is not the way to make new discoveries, it does seem to be a preferred way to get ahead in much of today's world.

Permalink to Comment

13. Yggdrasil on July 6, 2010 4:45 PM writes...

On the difficulties of publishing comments through the traditional letter to the editor route, see Prof. Rick Trebino's "How to Publish A Scientific Comment in 123 Easy Steps": http://www.scribd.com/doc/18773744/How-to-Publish-a-Scientific-Comment-in-1-2-3-Easy-Steps

Permalink to Comment

14. Hap on July 6, 2010 5:26 PM writes...

Except bad data is worse than no data, because people think that since the data has been published, the hypotheses of note have been tested and either supported and found wanting, and in lots of cases, that simply isn't true. Working on the basis of bad data (if you do good work) might be useful or might be a complete waste, but is more likely to be the latter than the former. If you want people to get on with their work and cease criticizing that of others, the easiest way to do that is to make the review process reasonably airtight, so that people can trust that what is published is real (or at least doesn't have glaring flaws), and then other people can decide what to do with it (and actually do something, instead of having to figure out whether it is real before they dare to waste their time and money working on it).

Permalink to Comment

15. Carl Lumma on July 6, 2010 5:53 PM writes...

There is also:

http://www.naboj.com

for arxiv and PubMed comments. -C.

Permalink to Comment

16. TFox on July 6, 2010 7:13 PM writes...

Part of it is cultural, I think, with most commentary being well handled by the existing mechanisms of private correspondence with the authors and formally reviewed published comments. Journals are not like a blog, where encouraging off-the-cuff reactions is sometimes the point. That said, BMJ seems to have been successful in building readership and writership for their nonreviewed Rapid Responses section.

Permalink to Comment

17. Anonymous on July 6, 2010 8:23 PM writes...

With all due respect, you guys don't know what you are talking about. The scientific literature is totally and completely mired in crap, and not a day goes by that I don't read a paper that I couldn't basically invalidate to an interested reader with a few critical sentences. Most people I know feel the same way.

Yet no one does this. Why not? Is it really because "the system works," or because there are other mechanisms for criticism? No. Academic scientists are every day of their lives perched on the edge of losing their funding or being unable to get their next paper accepted. Except at the government labs, there is NO meaningful job security (in the sense that your lab could disappear at any time). Criticizing the work of others makes it more likely that your grants will be harder to get, your papers harder to get accepted. In other words, it makes it more likely that your career will end.

So, you ask why no one ever comments at the Nature and other websites? Because there is only a downside to criticize the work of others, no upside. Read, chuckle, and move on. I do it at least five times a day.

Permalink to Comment

18. Malcolm on July 6, 2010 10:45 PM writes...

I've seen this work well in online medical journals, particularly the British Medical Journal's 'Rapid responses' feature.

An added nicety is that the editors pick up some of the better comments and re-publish them as 'letters to the editor'.

Here's an example from the BMJ (hopefully accessible to non-subscribers): http://www.bmj.com/cgi/eletters/333/7572/791

Permalink to Comment

19. Jack Bauer on July 7, 2010 2:09 AM writes...

Half the comments for synthetic papers would be "Reactions don't work as advertised."

Permalink to Comment

20. fungus on July 7, 2010 3:27 AM writes...

I think it's because the vast majority of people don't care enough to comment. Look at Totally Synthetic. Beautiful papers, maybe 5 substantive comments. Then the rest are "Paul you made a typo", "this crap doesn't belong in JACS", "Baran could have done it better", "KC is an egotist" etc. Someone should develop a bingo card for the comments, they fall in very set patterns.

Permalink to Comment

21. Keith Grimaldi on July 7, 2010 3:51 AM writes...

It's a good question because the Rapid Responses section of the British Medical Journal has been very popular for many years. Maybe the journals should make it easier, encourage it

Permalink to Comment

22. MarkA on July 7, 2010 10:51 AM writes...

This is an intriguing question. Why is it that every profession fears to openly critique themselves and their colleagues. As for annonymity, it just opens the door for vulgar and defamatory comments w/o contributing to issues at hand. CafePharma is a prime example. Just go to any of the forums - especially the company specific forums - and you'll quickly see its simply a site for pharma sales reps to vent.

http://www.cafepharma.com/

Permalink to Comment

23. Scott Kern on July 9, 2010 1:33 PM writes...

For biomedical publications, the website for comments has been operating for a few years. It's BioMed Critical Commentary, found at bm-cc.org. It has hundreds of comments and on occasion has induced a journal to request a retraction. BMCC was highlighted in the NetWatch section of the journal Science, 21 November 2008 322: 1169. What's more interesting is the tendency of scientists to ignore contrary data and commentary. Commentary is, indeed, under-appreciated.

Permalink to Comment

24. anon on December 9, 2011 1:46 PM writes...

pubmedreview.com

Permalink to Comment

25. cheapscrips on June 23, 2014 3:25 PM writes...

Aw, this was a very good post. Taking the time and actual effort to produce a top notch article… but what can I say… I procrastinate a lot and don’t manage to get anything done.

Permalink to Comment

26. Mugshot Removal on July 15, 2014 7:49 AM writes...

I could not refrain from commenting. Well written!

Permalink to Comment

POST A COMMENT




Remember Me?



EMAIL THIS ENTRY TO A FRIEND

Email this entry to:

Your email address:

Message (optional):




RELATED ENTRIES
Neratinib Comes Through For Puma
How Many Biopharma Employees Would Rather Be Working Somewhere Else?
The Broad Gets $650 Million For Psychiatric Research
Put Them in Cells and Find Out
The Hep C Field Gets Nastier By the Minute
Allergan Twists and Turns
Chemistry Class For the "Food Babe"?
Thalidomide, Bound to Its Target