About this Author
DBL%20Hendrix%20small.png College chemistry, 1983

Derek Lowe The 2002 Model

Dbl%20new%20portrait%20B%26W.png After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases. To contact Derek email him directly: Twitter: Dereklowe

Chemistry and Drug Data: Drugbank
Chempedia Lab
Synthetic Pages
Organic Chemistry Portal
Not Voodoo

Chemistry and Pharma Blogs:
Org Prep Daily
The Haystack
A New Merck, Reviewed
Liberal Arts Chemistry
Electron Pusher
All Things Metathesis
C&E News Blogs
Chemiotics II
Chemical Space
Noel O'Blog
In Vivo Blog
Terra Sigilatta
BBSRC/Douglas Kell
Realizations in Biostatistics
ChemSpider Blog
Organic Chem - Education & Industry
Pharma Strategy Blog
No Name No Slogan
Practical Fragments
The Curious Wavefunction
Natural Product Man
Fragment Literature
Chemistry World Blog
Synthetic Nature
Chemistry Blog
Synthesizing Ideas
Eye on FDA
Chemical Forums
Symyx Blog
Sceptical Chymist
Lamentations on Chemistry
Computational Organic Chemistry
Mining Drugs
Henry Rzepa

Science Blogs and News:
Bad Science
The Loom
Uncertain Principles
Fierce Biotech
Blogs for Industry
Omics! Omics!
Young Female Scientist
Notional Slurry
Nobel Intent
SciTech Daily
Science Blog
Gene Expression (I)
Gene Expression (II)
Adventures in Ethics and Science
Transterrestrial Musings
Slashdot Science
Cosmic Variance
Biology News Net

Medical Blogs
DB's Medical Rants
Science-Based Medicine
Respectful Insolence
Diabetes Mine

Economics and Business
Marginal Revolution
The Volokh Conspiracy
Knowledge Problem

Politics / Current Events
Virginia Postrel
Belmont Club
Mickey Kaus

Belles Lettres
Uncouth Reflections
Arts and Letters Daily
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline

In the Pipeline

« Palladium Couplings: You Can't Run Them All | Main | Is Cancer A Disease of the Modern World? »

October 19, 2010

Trusting the Medical Literature?

Email This Entry

Posted by Derek

How reliable is the medical literature, anyway? This profile of John Ioannidis at The Atlantic is food for thought. Ioannidis is the man behind the famous "Why Most Published Medical Findings Are False" paper a few years ago, and many others in the same vein.

The problems are many: publication bias (negative findings don't get written up and reported as often), confirmation bias, and desire to stand out/justify the time and money/get a grant renewal. And then there's good old lack of statistical power. Ioannidis and his colleagues have noted that far too many studies that appear in the medical journals are underpowered, statistically, relative to the claims made for them. The replication rates of such findings are not good.

Interestingly, drug research probably comes out of his analysis looking as good as anything can. A large confirmatory Phase III study is, as you'd hope, the sort of thing most likely to be correct, even given the financial considerations involved. Even then, though, you can't be completely sure - but contrast that with a lot of the headline-grabbing studies in nutrition or genomics, whose results are actually more likely to be false than true.

Ioannidis's rules from that PLoS Medicine paper are worth keeping in mind:

The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.

The smaller the effect sizes in a scientific field, the less likely the research findings are to be true.

The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true.

The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true.

The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.

The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.

And although he's talking about the published literature, these things are well worth keeping in mind when you're looking at your own internal data in a drug discovery project. Some fraction of what you're seeing is wrong.

Comments (17) + TrackBacks (0) | Category: Clinical Trials | Drug Development | The Scientific Literature


1. pete on October 19, 2010 12:50 PM writes...

Readers lament: "Oh, if only questionable articles could be 100% PATENTLY WRONG." The bitch is that more often they're a variable mix of truth and crap.

Permalink to Comment

2. lewis robinson on October 19, 2010 1:56 PM writes...

God yes ! I had to read the medical literature extensively to keep up professionally for years and years. For a few horrible examples see

Things are looking up however. A math prof I audited a course from a few years ago yesterday told me that premeds wanting to get in to Harvard Med School will now have to take and pass a course in statistics. The rest are fairly certain to follow Harvard's lead.

The prof is panicked because there are apparently more jobs for statisticians in industry and elsewhere than there are statisticians. He urged me to have a violinist friend who is a biostatistician working for a large medical center send him her CV.

This will be somewhat helpful, but just know statistics doesn't address the main problem -- how to logically dissect a scientific paper (or a paper purporting to be scientific) a process which often has nothing to do with the statistics involved.

Again, see the link for two examples where this wasn't done.

Permalink to Comment

3. Maks on October 19, 2010 2:49 PM writes...

Ioannidis seems to have some ego issues. All his articles are virtually repeating the same argument again and again, just in different journals. He's complaining about self citations and cites himself 5 out of 24 citations, the article "Why Most Published Medical Findings Are False" is cherry picking on statistics and throws different disciplines in one pot just to prove his point. This guy is more part of the problem of modern science than a solution.

Permalink to Comment

4. Elaine Schattner, M.D. on October 19, 2010 4:20 PM writes...

The "list of 10" is a bit too simple, but the point - that much of what gets published in medicine is incorrect - is really important. Doctors and scientists who serve as editors of highly-rated journals may be eager to push through "important" researchers' findings, and peer reviewers may be eager to "help" colleagues who, in turn, might review their grants and write letters for promotion.

So it would be an academic medical game, except that doctors and patients make important decisions based on what's published, funded and cited.

Permalink to Comment

5. Resveratrol Receptor on October 19, 2010 6:15 PM writes...

How do we know that Ioannidis's research about research being false isn't false?

Permalink to Comment

6. Jose on October 19, 2010 6:41 PM writes...

I am glad you single out genomics and nutrition. Essentially no large GWAS study has duplicated in another population, and nutrition surveys are staggeringly difficult. How many times per week on average have you had green tea in the past 5 years? How strong? How big was the cup? The error bars get really huge really fast.

Permalink to Comment

7. drug_hunter on October 19, 2010 8:33 PM writes...

My all-time favorite set of rules for how to detect schlock and delusion (as opposed to outright fraud) is found in Langmuir's 1953 lecture on "pathological science" --

Permalink to Comment

8. Pallas Renatus on October 19, 2010 9:09 PM writes...

I have to agree with #6 above. Not to mention the huge variability in the typically survey-based statistics they run.

Permalink to Comment

9. London_Chemist on October 20, 2010 2:20 AM writes...

Probably stating the obvious, but quite a few of these apply to ALL areas of scientific research (especially the last few). Just think of all the fuss about global warming data and research we've witnessed this year.....

Permalink to Comment

10. Cartesian on October 20, 2010 4:21 AM writes...

"The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true."

Yes it can push some to cheat, but like in sport normally the truth will win; because even if some do take some stimulating products, normally they are detected and the real winner will have the place. Otherwise the fact that there is nothing to earn with a job will not push to do it well, and like this there will not be the best truth ; because even if there is money this is sure that if you have the truth, you are in a better position in order to have the money, the truth is still of a great value; then the result should be better with money, if some rules are correctly respected.

Permalink to Comment

11. alig on October 20, 2010 8:00 AM writes...

@ #10,

Yeah, because ARod and all the other cheaters gave back all the money the got from breaking the rules. Cheating is profitable. So how did truth win in sport?

Permalink to Comment

12. Lacerta Bio on October 20, 2010 8:21 AM writes...

Even worse, we're not even sure what a placebo is:

Permalink to Comment

13. Al on October 20, 2010 10:07 AM writes...

What percentage of reported synthetic yields, ee's, etc., reported in JACS do you suppose are repeatable?

Successful experiments get published, failed ones usually don't (unless there is conclusive proof about the reason for the failure).

Permalink to Comment

14. Mr Spock on October 20, 2010 10:06 PM writes...

#3 Maks writes: Ioannidis seems to have some ego issues

Argumentum as hominen is a logical fallacy.

Permalink to Comment

15. Cartesian on October 21, 2010 3:30 AM writes...

To 11 Alig : Is that because some criminals are not arrested that crime is winning ? But some of the victimes do still deserve to be paid (even if they have been stolen some money), and some would not even do their job if there were no money to earn. Also I think Derek prefers to be paid, and it is better for his family as well.

Permalink to Comment

16. gippgig on October 28, 2010 6:07 PM writes...

A review in the Washington Post newspaper of the book "Bad Science" by Ben Goldacre states, "Goldacre shines in a chapter about bad scientific studies by writing it from the perspective of a make-believe big pharma researcher who needs to bring a mediocre new drug to market. He explains exactly how to skew the data to show a positive result."

Permalink to Comment

17. loupgarous on January 31, 2012 10:29 PM writes...

In other words, the bigger the p-values, the less likely the article's assertions are to be true. I realize Ioannidis had to cut this up into bite-size pieces for the audience of the Atlantic, but this is no news to anyone with a background in biotech.

Irving Langmuir's landmark essay "Pathological Science" would have been just as telling, though. He covered almost the same territory sixty years ago.

Permalink to Comment


Remember Me?


Email this entry to:

Your email address:

Message (optional):

The Last Post
The GSK Layoffs Continue, By Proxy
The Move is Nigh
Another Alzheimer's IPO
Cutbacks at C&E News
Sanofi Pays to Get Back Into Oncology
An Irresponsible Statement About Curing Cancer
Oliver Sacks on Turning Back to Chemistry