Corante

About this Author
DBL%20Hendrix%20small.png College chemistry, 1983

Derek Lowe The 2002 Model

Dbl%20new%20portrait%20B%26W.png After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases. To contact Derek email him directly: derekb.lowe@gmail.com Twitter: Dereklowe

Chemistry and Drug Data: Drugbank
Emolecules
ChemSpider
Chempedia Lab
Synthetic Pages
Organic Chemistry Portal
PubChem
Not Voodoo
DailyMed
Druglib
Clinicaltrials.gov

Chemistry and Pharma Blogs:
Org Prep Daily
The Haystack
Kilomentor
A New Merck, Reviewed
Liberal Arts Chemistry
Electron Pusher
All Things Metathesis
C&E News Blogs
Chemiotics II
Chemical Space
Noel O'Blog
In Vivo Blog
Terra Sigilatta
BBSRC/Douglas Kell
ChemBark
Realizations in Biostatistics
Chemjobber
Pharmalot
ChemSpider Blog
Pharmagossip
Med-Chemist
Organic Chem - Education & Industry
Pharma Strategy Blog
No Name No Slogan
Practical Fragments
SimBioSys
The Curious Wavefunction
Natural Product Man
Fragment Literature
Chemistry World Blog
Synthetic Nature
Chemistry Blog
Synthesizing Ideas
Business|Bytes|Genes|Molecules
Eye on FDA
Chemical Forums
Depth-First
Symyx Blog
Sceptical Chymist
Lamentations on Chemistry
Computational Organic Chemistry
Mining Drugs
Henry Rzepa


Science Blogs and News:
Bad Science
The Loom
Uncertain Principles
Fierce Biotech
Blogs for Industry
Omics! Omics!
Young Female Scientist
Notional Slurry
Nobel Intent
SciTech Daily
Science Blog
FuturePundit
Aetiology
Gene Expression (I)
Gene Expression (II)
Sciencebase
Pharyngula
Adventures in Ethics and Science
Transterrestrial Musings
Slashdot Science
Cosmic Variance
Biology News Net


Medical Blogs
DB's Medical Rants
Science-Based Medicine
GruntDoc
Respectful Insolence
Diabetes Mine


Economics and Business
Marginal Revolution
The Volokh Conspiracy
Knowledge Problem


Politics / Current Events
Virginia Postrel
Instapundit
Belmont Club
Mickey Kaus


Belles Lettres
Uncouth Reflections
Arts and Letters Daily
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline

In the Pipeline

« Is Ampyra Any Good? | Main | A Quick Tour Through Drug Development Reality »

August 14, 2012

Reproducing Scientific Results - On Purpose

Email This Entry

Posted by Derek

We've spoken several times around here about the problems with reproducing work in the scientific literature. You have to expect some slippage on cutting-edge work, just because it's very complex and is being looked at for the first time. But at the same time, it's that sort of work that we're depending on to advance a field, so when it turns out to be wrong, it causes more damage than something older and more obscure that falls apart.

There's a new effort which is trying to attack the problem directly. Very directly. The Reproducibility Initiative is inviting people to have their work independently confirmed by third-party researchers. You'll be responsible for the costs, but at the end of it, you'll have a certification that your results have been verified. The validation studies themselves will be published in the new PLOS ONE Reproducibility Collection, and several leading publishers have agreed to link the original publications back to this source.

I very much hope that this catches on. The organizers have rounded up an excellent advisory committee, with representatives from academia and industry, both of whom would be well served by more accurate scientific publication. I can especially see this being used when someone is planning to commercialize some new finding - going to the venture capital folks with independent verification will surely count for a lot. Granting agencies should also pay attention, and reward people accordingly.

Here's an article by Carl Zimmer with more on the idea. I'll be keeping a close eye on this myself, and hope to highlight some of the first studies to make it through the process. With any luck, this can become the New Normal for groundbreaking scientific results.

Comments (27) + TrackBacks (0) | Category: General Scientific News | The Scientific Literature


COMMENTS

1. weirdo on August 14, 2012 12:49 PM writes...

Nice start . . . but paying to have your own results reproduced? I don't see many takers, frankly.

Permalink to Comment

2. Cersei on August 14, 2012 12:55 PM writes...

Interesting idea, but I have my reservations about how well this can be carried out. In the end, what is the incentive for the third-party researchers to generate anything but a positive result? If they don't repeat the said data, the default reaction for the submitting author(s)would most likely be "Man, these guys are hacks! What a waste of money! Maybe we should have chosen this other CRO instead. They seem to be better at reproducing published findings". This is akin to an experience I had in graduate school where the the PI initially insisted that we send our samples to this one particular company because they were the best (i.e. they can generate the elemental analysis result that we needed for publication). Therefore, with the support of my supervisor, I decided to send these guys a fake sample. When the result came back "positive", we stopped sending compounds to them.

Permalink to Comment

3. Chemjobber on August 14, 2012 1:03 PM writes...

@2. Ms. Lannister -- Very clever ruse.

Permalink to Comment

4. SN on August 14, 2012 1:28 PM writes...

I've had a fair bit of first and second-hand experience with checking procedures for Organic Syntheses. Even though the submitting authors know their procedure needs to be reproduced by another lab prior to publication, it's seldom that things are reproduced (+/- 5% yield) on the first try. I wonder if the third party researchers are grad students or employed professionals.

Permalink to Comment

5. bacillus on August 14, 2012 2:01 PM writes...

This might work for organic synthesis, but it's a long shot for biomedical research involving animals. The average study that I publish probably costs >$200K if labour is included, take at least a year to perform, and require highly specialized facilities. I can't see me paying someone else to reproduce such studies any time soon.

Permalink to Comment

6. James on August 14, 2012 2:12 PM writes...

From a chemical education standpoint, I'd love for someone to publish a series of very boring SN1/SN2/E1/E2 reactions on alkyl halides, with yields and ratios included, so that we can point to actual data instead of waving our arms and talking ex recto about how much changing to a polar aprotic solvent will affect the amount of SN2 product.

Permalink to Comment

7. Boghog on August 14, 2012 2:31 PM writes...

This could really take off if the funding agencies directly paid for such studies. it would provide a mechanism for the agencies to directly check the reliability of research they are funding and also reduce potential conflicts of interest between the original PI and CRO that is doing the checking.

Permalink to Comment

8. MTK on August 14, 2012 2:45 PM writes...

Speaking of Org Syn, is there something similar in biology? Obviously experiments of the sort mentioned by bacillus (#5) wouldn't be applicable, but what about assays and other such things?

Permalink to Comment

9. J. Peterson on August 14, 2012 2:48 PM writes...

I also agree with @2...it seems like there's a strong conflict of interests if the "reproducers" are funded by the original research team.

Permalink to Comment

10. David Formerly Known as a Chemist on August 14, 2012 3:00 PM writes...

@7, Boghog...if funding agencies paid for such studies the paylines for funding grant applications would plummet below their already dismally low levels. Imagine how much of an agency's budget would be required to pay independent labs to reproduce research the agency already funded. Especially, as @5 Bacillus pointed out, biomedical research involving animal models costs hundreds of thousands of dollars.

Permalink to Comment

11. Boghog on August 14, 2012 3:24 PM writes...

@10, David ... The funding agencies would of course need to be very selective and check only very high profile results plus perhaps some random spot checks. Putting aside 10% of the budget for checking decrease the amount of funding for original research but will increase the impact and reliability of the remaining 90% that is still funded.

Permalink to Comment

12. mass_speccer on August 14, 2012 4:44 PM writes...

Org Syn is (in my opinion) absolutely amazing so anything that can expand that sort of idea would seem good. I'm not totally convinced by this model though, like others have pointed out reproducing some results would be incredibly expensive. The Org Syn model seems good but probably only works for (relatively) inexpensive procedures.

Maybe soon research grants will include money for research, publishing (open access), and reproducing results. This is going to cost a lot unfortunately.....

Permalink to Comment

13. Dave on August 14, 2012 6:21 PM writes...

Org Syn is great for reliability, and one of my fellow grad students has been working on submitting one for a while now. This Reproducibility Initiative sounds interesting, but at the end of the day, how much do you have to pay for the result validation? It seems like it's on a case-by-case basis. The cost might be a turn-off for most academics, and those working with cells & animals may be hesitant just because they already know that the results aren't always consistent.

Permalink to Comment

14. Stevo on August 14, 2012 6:50 PM writes...

Ive said it before, but probably the easiest way of getting rid of so many inflated and bad papers is to allow properly moderated online comments linked to the article for all paper. It would be a real deterent if you published a paper, only to find 50 people had replied to say it didnt work.

Permalink to Comment

15. Watson on August 14, 2012 7:10 PM writes...

Why not have CERN fund an independent validation of their Higgs results using the exact same methodology?

The risk of non-validated results and the associated costs of validation are bound to be the most important factors for something like this to work. There is only so much you can do about costs, so unless there was some sort of punishment related to not taking the risks, I don't see this practice making much of an impact.

It would be nice to know that we're not wasting our time on irreproducible results, though.

Permalink to Comment

16. Mat Todd on August 14, 2012 8:18 PM writes...

One of the benefits of open science, i.e. having one's lab book online and open to anyone, is that all raw data are reported, including for reactions that don't work, and including repeats of reactions that do work. One of our current ones is here: http://malaria.ourexperiment.org/tcmdc_ap

Lots of raw data + the option for anyone to criticize the data tends to make the science reproducible.

Permalink to Comment

17. Colonel Boris on August 14, 2012 9:49 PM writes...

While it's a nice idea, it would take a huge chunk of our budget to do this. The checking for my work would have to be spread over a few groups as we have a range of properties that we look for and a range of techniques we use (magnetism, gas sorption, molecular modelling, synchrotron X-ray studies, AFM measurements, etc) and it costs us enough to do these ourselves, let alone how long it takes to get some of our stuff to measurement.
Not only would it cost us, but I don't know too many other groups in the field who would want to take that much time out of their own research schedule to test others' work.
On the subject of time, if we were asked to re-synthesise something and the chemicals weren't on hand, it can take a minimum of three weeks for compounds to reach Australia if they're not in stock here already (and you can bet it won't be). Would researchers be happy to wait an extra month for approval of their manuscripts, especially if there were other journals of a similar impact that didn't require it?
I like the idea, honestly, but in practice I can't see it happening in all fields.

Permalink to Comment

18. Jose on August 14, 2012 11:45 PM writes...

Re: Steveo

http://chem.chem.rochester.edu/nvdcgi/mojo.cgi

"Literature procedures do not always give consistent results. Sometimes this is the fault of the chemist, and sometimes the fault of the chemistry. This list contains reaction chemistry from the literature that is capricious or difficult to reproduce. This page is a pure reflection of visitor input - it has never been edited internally (disclaimer: the opinions displayed here are not those of the site administrator)."

Personal fave is Benzylic Oxidation with IBX- do a SciFinder rx search on it. You'll be suprised (or not).

Permalink to Comment

19. Alex on August 15, 2012 1:33 AM writes...

What the fuck, that is the fucking question? What the fuck about rules? About expectations? Why do we try and try to get to the point where rich people are? To get everything everything? Money, fame, success? What the fuck, we don’t need that? Why is alcohol bad? Why? Because it makes you realize that is not worthy, that you don’t fucking fit in the system? The system is broke we are seeing it everyday in the news, why we try try try, try what, get papers? Get a position somewhere? Get a job? Who cares? Who is happy here? Who is really happy? You with the positions, you that are trying? Alcohol is not bad, it makes you realize what the important part of life is, to live, that is it, not to be successful, rich, what the fuck rich people are not happy, are we aware of that? That is just an assumption of us poor people. Rich, money, success is not the way, we should learn to enjoy, to thank more for what we have, and not to be sad about what we don’t have, it is a waste of time, not worthy, think about it during your next drink. Some alcohol is not bad, it just make you see what you can not see, reality, not this fucking system dream. Reality is man, you are just so lucky to be alive, don’t waste the time worry about bullshit, live your life don’t try to pretend; don’t try to get where others are. Cheers
I am just in water right now, no more alcohol so I guess going back to the freaking system. And what? That is the question? For a couple of hours I was happier than you, you know that right? I don’t mean that you need to drink to be happy, the only thing that you need to do, is to realize that is not worthy to waste your time worrying about everything, a few drinks was my way but tomorrow I promise my day will be different, of course it will be, little by little I will realize that that is not the point of life, no way, be happy, enjoy, it is too short. Regards to everybody, academia, industry and hell. Cheers.

Permalink to Comment

20. Will on August 15, 2012 8:08 AM writes...

@13 Dave

I don't know if you realize it, but your last sentence is the whole point of the issue. If something is not reproducible, in an ideal world it would either not be ready for publication, or would be published in such a way that alerted the reader to the reproducibility problems.

Permalink to Comment

21. Helical_Investor on August 15, 2012 11:02 AM writes...

Hmmmm ... it is nice in theory, but in practice probably too cumbersome. I think some journals might do well requiring it. It would serve a simpler topic set like tetrahedron letters well, but would be a hang up for Nature and Science type publications. I would certainly not want to be in a 'race' on any scientific cutting edge topic and have to wait on verification (especially if I was trying to finalize the degree : ).

So, limited practical value in my opinion. I do like moderated comments for journal publications as a more workable consideration.

Zz

Permalink to Comment

22. newnickname on August 15, 2012 11:58 AM writes...

I've born the brunt of not being able to reproduce the claims of quite a few papers. Some have had yield problems (tolerable, without more practice); some produced the wrong product!; some didn't work at all; some seemed too good to be true.

If I had the money, there are quite a few that I would pay to have tested just to be vindicated!

If the proposed reproducibility test were to be implemented, it should require publication of the results, confirmatory OR contradictory.

I agree with @14 Stevo: Allow moderated on-line comments on e-pubs. The article authors are NOT to be the moderators! Comments would not be limited to experimental procedures, but design, implementation and interpretation as well as proper citation of relevant precedents. I guess I might as well as say "everything scientifically and ethically relevant".

Permalink to Comment

23. newnickname on August 15, 2012 12:04 PM writes...

Corrections / clarification of my post above:
1. bornE the brunt
2. several of the papers I couldn't reproduce were subsequently reported to be unreliable by others.
3. there are still several others I'd like to see exposed.

Permalink to Comment

24. jojo on August 16, 2012 11:10 AM writes...

Can we pay to have other people's work reproduced without their consent? I'd be totally down (You know who your are)

Permalink to Comment

25. Anonymous on August 16, 2012 3:24 PM writes...

reproduction is a glorified concept in modern science.
the boiling point of water is reproducible, an nmr of the same compound at the same purity is reproducible.
but in a time when training of undergrads varies in between countries and native english speakers are a minority among publishing researchers preparative procedures are hardly in the same category as the first two examples. in a lays words: preparative chemistry is not lego.
high impact publications often combine highly specialised work and massively oversold findings, what is this initiative about? buying a citation from people who back this kind of research for a PLOS paper and funding? quite idealistic and ambitious, yet unrealistic in my opinion.

Permalink to Comment

26. wlm on August 19, 2012 4:13 PM writes...

I think the funding agencies should have a mechanism (via expert panels, surveys of grantees, etc) to fund reproduction of select findings. It shouldn't be up to the labs which originally published the results. I think it would be well worth the money to try to reproduce even the most accessible results (not necessarily all the results) from the most influential 0.1% of publications each year.

Similarly, they should have a mechanism for reporting publications lacking crucial data or methods.

Permalink to Comment

27. Bryan Sanctuary on October 7, 2012 11:46 AM writes...

Sure people make errors and they permeate all of science. Sometime errors take years to uncover. For example one of the greatest mathematicians of the 20th century, von Neumann, "proved" that quantum mechanics was complete and it took 40 years to find the error.

If I look at the number of pages in Journals today they have exploded since I was a graduate student. It is impossible to reproduce even a fraction of the work, much of which is probably wrong.

In contrast, if something is really important it won't be long before people start to use it and, pretty fast, any irreproducible results will emerge.

If you want to put a lot of money into something, then of course you need to be careful. For example imagine if we could teleport? In fact the teleportation work is being considered for the Nobel Prize in Physics this year. Well it will never work because it is fundamentally flawed, but money keeps pouring into it.

So I say let the system be. Ideas come into vogue and fall out. Polymers are now replaced by self-assembly. I say simply peer review, put out the stuff and let the dust settle down into results and techniques that pass the test of time.

On the other hand, if we talk about Pharma companies, well I want those pills tested well and side effects documented. I abhor off-labeling and other unhanded ways of misleading the public on issues of help. So FDA and other watch dog institutions must make sure that the data is reproducible and correct.

Permalink to Comment

POST A COMMENT




Remember Me?



EMAIL THIS ENTRY TO A FRIEND

Email this entry to:

Your email address:

Message (optional):




RELATED ENTRIES
The Worst Seminar
Conference in Basel
Messed-Up Clinical Studies: A First-Hand Report
Pharma and Ebola
Lilly Steps In for AstraZeneca's Secretase Inhibitor
Update on Alnylam (And the Direction of Things to Come)
There Must Have Been Multiple Chances to Catch This
Weirdly, Tramadol Is Not a Natural Product After All