Corante

About this Author
DBL%20Hendrix%20small.png College chemistry, 1983

Derek Lowe The 2002 Model

Dbl%20new%20portrait%20B%26W.png After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases. To contact Derek email him directly: derekb.lowe@gmail.com Twitter: Dereklowe

Chemistry and Drug Data: Drugbank
Emolecules
ChemSpider
Chempedia Lab
Synthetic Pages
Organic Chemistry Portal
PubChem
Not Voodoo
DailyMed
Druglib
Clinicaltrials.gov

Chemistry and Pharma Blogs:
Org Prep Daily
The Haystack
Kilomentor
A New Merck, Reviewed
Liberal Arts Chemistry
Electron Pusher
All Things Metathesis
C&E News Blogs
Chemiotics II
Chemical Space
Noel O'Blog
In Vivo Blog
Terra Sigilatta
BBSRC/Douglas Kell
ChemBark
Realizations in Biostatistics
Chemjobber
Pharmalot
ChemSpider Blog
Pharmagossip
Med-Chemist
Organic Chem - Education & Industry
Pharma Strategy Blog
No Name No Slogan
Practical Fragments
SimBioSys
The Curious Wavefunction
Natural Product Man
Fragment Literature
Chemistry World Blog
Synthetic Nature
Chemistry Blog
Synthesizing Ideas
Business|Bytes|Genes|Molecules
Eye on FDA
Chemical Forums
Depth-First
Symyx Blog
Sceptical Chymist
Lamentations on Chemistry
Computational Organic Chemistry
Mining Drugs
Henry Rzepa


Science Blogs and News:
Bad Science
The Loom
Uncertain Principles
Fierce Biotech
Blogs for Industry
Omics! Omics!
Young Female Scientist
Notional Slurry
Nobel Intent
SciTech Daily
Science Blog
FuturePundit
Aetiology
Gene Expression (I)
Gene Expression (II)
Sciencebase
Pharyngula
Adventures in Ethics and Science
Transterrestrial Musings
Slashdot Science
Cosmic Variance
Biology News Net


Medical Blogs
DB's Medical Rants
Science-Based Medicine
GruntDoc
Respectful Insolence
Diabetes Mine


Economics and Business
Marginal Revolution
The Volokh Conspiracy
Knowledge Problem


Politics / Current Events
Virginia Postrel
Instapundit
Belmont Club
Mickey Kaus


Belles Lettres
Uncouth Reflections
Arts and Letters Daily
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline

In the Pipeline

« Return of the Arsenic Bacterium | Main | Another Two-Person Drug Company »

June 2, 2011

Biomarkers, Revisited. Unfortunately.

Email This Entry

Posted by Derek

Your genome - destiny, right? That's what some of us thought - every disease was going to have one or more associated genes, those genes would code for new drug targets, and we'd all have a great time picking them off one by one. It didn't work out that way, of course, but there are still all these papers out there in the literature, linking Gene A with the chances of getting Disease B. So how much are those worth?

While we're at it, everyone also wanted (and still wants) biomarkers of all kinds. Not just genes, but protein and metabolite levels in the blood or other tissue to predict disease risk or progression. I can't begin to estimate how much work has been going into biomarker research in this business - a good biomarker can clarify your clinical trial design, regulatory picture, and eventual marketing enormously - if you can find one. Plenty of them have been reported in the literature. How much are those worth, too?

Not a whole heck of a lot, honestly, according to a new paper in JAMA by John Ioannidis and Orestes Panagiotou. They looked at the disease marker highlights from the last 20 years or so, the 35 papers that had been cited at least 400 times. How good do the biomarkers in those papers have to be to be useful? An increase of 35% in the chance of getting the targeted condition? Sorry - only one-fifth of the them rise to that level, when you go back and see how they've held up in the real world.

Subsequent studies, in fact, very rarely show anything as strong as the original results - 29 of the 35 biomarkers show a less robust association after meta-analysis of all the follow-up reports, as compared to what was claimed at first. And those later studies tend to be larger and more powered - in only 3 cases was the highly cited study the largest one that had been run, and only twice did the largest study show a higher effect measure than the original highly cited one. Only 15 of the 35 biomarkers were nominally statistically significant in the largest studies of them.

Ioannidis has been hitting the literature's unreliability for some time now, and I think that it's hard to dispute his points. The first thought that any scientist should have when an interesting result is reported is "Great! Wonder if it's true?" There are a lot of reasons for things not to be (see that earlier post for a discussion of them), and we need to be aware of how often they operate.

Comments (25) + TrackBacks (0) | Category: Biological News | The Scientific Literature


COMMENTS

1. RespiSci on June 2, 2011 9:43 AM writes...

In COPD it has been a real struggle to identify valid biomarkers. No doubt, there is an enormous advantage for your drug if you can show an effect on a valid biomarker especially after a short-term administration of the drug (during early Phase 2 studies for example). The question is always which biomarker to select. We just don't know. There are always reports of trends, but when dealing with a limited budget, you can't go on a fishing expedition to look at a plethora of biomarkers. And what do you measure- gene expression or protein levels or both?

Perhaps instead of trying to get into drug development, the NIH could focus on validating biomarkers for various diseases? Would that be a better use of resources and funds?

Permalink to Comment

2. luysii on June 2, 2011 9:58 AM writes...

The ultimate biomarker is a genetic variant. It's stretching things a bit to say that the sickle gene is a biomarker for sickle cell anemia, but large efforts have been made to search for genetic risk factors for various diseases, particularly psychiatric disease. The literature is wildly inconsistent with many failures of replication.

A possible explanation for this is the differential effect of the environment on a genetic variant. For example, two polymorphisms in the gene for the serotonin reuptake protein (the target of the SSRIs) have been associated with an increased risk of depression.

A recent study [ Proc. Natl. Acad. Sci. vol. 108 pp. 8189 - 8193 '11 ] looked at two SSRI polymorphisms and the risk of postpartum depression (rather loosely defined). They found if a woman had 3 or 4 of the 'high risk' markers her chances of postpartum depression were determined by her social class.

If she was of low socioeconomic status (SES) -- as determined by her educational level (yet another type of marker) the risk of postpartum depression was increased, while if she was of high SES, the risk was DECREASED.

Fascinating, no? Could this sort of thing, if widespread, explain failure of replication in marker studies?

For more detail, see the study or see https://luysii.wordpress.com/2011/05/31/a-seminal-paper-if-the-conclusions-follow-from-the-actual-data-which-i-cant-find/

Permalink to Comment

3. Hap on June 2, 2011 10:20 AM writes...

Yes, it would, but the NIH seems uninterested. It might be something for a consortium of drug companies, though - they have a direct stake in knowing what biomarkers work and convincing the FDA that they do so that they can run better (and potentially cheaper) trials.

Permalink to Comment

4. johnnyboy on June 2, 2011 11:33 AM writes...

"Biomarkers" is a very broad term. You're focusing on the presence of genes or gene mutations as possible predictors of disease, which to me is way, way out there on the fringes of what biomarkers can and should be. There are probably hundreds of blood and urine-based biomarkers which work just fine in their intended use, ie. to provide a quick and easy indication of disease presence or evolution. None of them are perfect indicators, but the people who use them on a daily basis know their limitations well, and if they have survived the test of time it's because their benefits outweigh their limitations. To dismiss the concept of biomarkers based on a statistical study that includes a whopping (count'em) 35 articles, a study which from its abstract doesn't appear to draw distinctions between the different types and uses of biomarkers, is throwing the baby, the bath, and the whole damn plumbing out with the bathwater.

Permalink to Comment

5. bacillus on June 2, 2011 11:51 AM writes...

I agree with jonnyboy@3 that a broader definition of biomarkers reveals that they are not just a pipedream E.g. many antibody responses are used as biomarkers of successful vaccine take or as correlates of protection. Likewise elevated liver enzymes in serum correlate well with liver damage.

P.S. only discovered this blog a few weeks ago, and I love it, though I'm a microbiologist rather than a chemist.

Permalink to Comment

6. cynical1 on June 2, 2011 12:31 PM writes...

@luysii (#2)......... I think the biomarker they found was that money really can buy happiness.

Permalink to Comment

7. luysii on June 2, 2011 1:16 PM writes...

@ cynical1 (#6) -- Damn, I wish I'd been smart enough to think of that as a title for the post !

Permalink to Comment

8. RespiSci on June 2, 2011 1:30 PM writes...

@luysii (#2) as johnnyboy (#4)mentioned, we aren't just interested in genetic variants to let us know if a patient should be treated with drug X or Y, but we are looking for early indicators that the drug is working.

@johnnyboy, I don't know of hundreds of biomarkers that have been validated for a disease. For our indication of COPD, improvement of lung function takes 6 months-1 year to demonstrate. How we would love to have a blood or urine biomarker that if changed (up or down, I don't care!) in weeks or under 3 months to let us know if the drug is working. But for every biomarker that is the flavour of the month, they don't seem to hold up under more robust and larger studies.

@Hap (#3)ah... a consortium of drug companies....Have been trying to put one in for another type of question and it is slow going but not impossible!

Permalink to Comment

9. William B Swift on June 2, 2011 2:10 PM writes...

I'm only an interested amateur, but I have to admit to being surprised that anyone really expected much success from such (relatively) simple tests or flags. Human physiology makes the weather look like an incredibly simple system; with all of the interactions between genes and with the environment I would have been amazed if there were many useful or effective predictions coming out of this research already.

Permalink to Comment

10. TJMC on June 2, 2011 2:50 PM writes...

Hap and Repisci - there are at least two biomarker consortiums that have been around for several years. One at Anderson Cancer Center, the other based in DC. I agree that many older "discoveries" do not hold up to broader trials. Still, the discipline of finding and validating them (to a point acceptable to the FDA as clinical surrogates) is maturing. So I would especially expect older papers to fail in holding up. I think you will find more recent data on biomarkers are actively being added to (public) databases and less so in journals.

Some view a biomarker as a competitive advantage, but it becomes a double-edged sword in Pharma. Whatever time advantage it offers, it still has to be submitted as part of a public proof to the regulators of efficacy. Regulators are loath to accept anything except a broad acceptance from outside disease experts during a long POV discussion. So that advantage is then lost.

I helped host several biomarker summits. The summary from the last one is here and touches on much of the above post and comment issues. Figure 6 from H. Young of AstraZeneca is especially helpful for understanding the wide and evolving role of biomarkers: http://bit.ly/mwymir

Permalink to Comment

11. Josh on June 2, 2011 4:13 PM writes...

@#6
But happiness can't buy you money.

Permalink to Comment

12. Luysii on June 2, 2011 6:36 PM writes...

#8 RespiSci -- Agree that genetic variants are not a biomarker. I used CPK as a biomarker to follow patients with polymyositis and to do carrier testing for Duchenne muscular dystrophy (before we had the gene), and the sedimentation rate to follow patients with temporal arteritis.

Nonetheless, the PNAS paper is quite relevant (if true) to the failure of replication of biomarker studies, as I doubt that any of them take into account socioeconomic status or life experiences of their subjects.

The notion that a genetic variant could be a risk factor for one group of people and a protective factor for another is truly staggering, and (if true) a paradigm shifter.

Permalink to Comment

13. TJMC on June 2, 2011 10:41 PM writes...

Hap and Repisci - there are at least two biomarker consortiums that have been around for several years. One at Anderson Cancer Center, the other based in DC. I agree that many older "discoveries" do not hold up to broader trials. Still, the discipline of finding and validating them (to a point acceptable to the FDA as clinical surrogates) is maturing. So I would especially expect older papers to fail in holding up. I think you will find more recent data on biomarkers are actively being added to (public) databases and less so in journals.

Some view a biomarker as a competitive advantage, but it becomes a double-edged sword in Pharma. Whatever time advantage it offers, it still has to be submitted as part of a public proof to the regulators of efficacy. Regulators are loath to accept anything except a broad acceptance from outside disease experts during a long POV discussion. So that advantage is then lost.

I helped host several biomarker summits. The summary from the last one is here and touches on much of the above post and comment issues. Figure 6 from H. Young of AstraZeneca is especially helpful for understanding the wide and evolving role of biomarkers: http://bit.ly/mwymir

Permalink to Comment

14. Zak on June 3, 2011 3:32 AM writes...

My wife is a doctor, and won't make a reservation for me at her hospital to get a checkup (I'm 38), because she says they'll just scan my blood for a bunch of biomarkers that have absolutely no clinical significance.

Permalink to Comment

15. BS on June 3, 2011 3:58 AM writes...

To expect that singular proteins or metabolites can be all encompassing predictive biomarkers beyond more serious conditions (liver damage, renal failure etc with some exceptions) or to expect that one markers fits all phenotypes is to seriously misunderstand the nature of biology, or rather to confuse the concepts of variance (population quality) with variability (individual quality reflecting reaction norms, crytic genetic variation, epigenotypes). The search for predictive SNPs is the most silly extreme of this delusion with geneticists wasting enormous amounts of cash on GWAS, money that could be used for something better.

Permalink to Comment

16. Still Scared of Dinosaurs on June 3, 2011 4:04 AM writes...

Basic stats tell us that 35 articles are a lot if they are (1) the 35 most cited articles in the field and (2) show a strong tendency to fail to replicate. Ioannidis isn't throwing out the concept of biomarkers, just making the point (again) that the quality of the average scientific publication is a lot lower than people think.

Permalink to Comment

17. dearieme on June 3, 2011 4:38 AM writes...

"All medical research is rubbish" is a better approximation to the truth than almost all medical research. Alas.

Permalink to Comment

18. Virgil on June 3, 2011 7:55 AM writes...

@1 & @3, I disagree the NIH is uninterested. Just search the NIH Guide for RFAs with "biomarker" in the title, and you'll see they're soliciting plenty of applications for all kinds of diseases.

The big issue I think, is knowing whether the biomarker is specific for a particular organ. The big one in MI for many years was creatine kinase in the blood, but then it turned out all kinds of muscle disease can give a false positive. Thankfully now we have cardiac troponin I which is more heart-specific. Of course, it is arguable that we don't really need a biomarker for MI - when someone drops to the floor complaining of chest pain its a pretty good bet they're having an MI.

I would argue that the marker field has focused too heavily on blood in the past, and I'd like to see more "behavioral" biomarkers - things such as a particular type of intelligence test that could detect early onset Alzheimer's, or something about posture/movement (detected with a kinect type of interface) that would detect early osteoporosis, etc.

Permalink to Comment

19. RespiSci on June 3, 2011 8:23 AM writes...

@TJMC (#10) I agree that the field of biomarkers is maturing but that the pace is slow. It is challenging when designing the clinical studies to determine which biomarkers you will measure, even as exploratory endpoints. When you fail to show an effect on a biomarker the knee jerk reaction is that the drug is ineffective and not that the wrong biomarker was selected.
Many thanks for the link and the info on the consortiums. (Derek, a huge bonus for us who follow your blog to share info like this)

Permalink to Comment

20. Jonathan on June 3, 2011 2:35 PM writes...

Yes, it would, but the NIH seems uninterested. It might be something for a consortium of drug companies, though - they have a direct stake in knowing what biomarkers work and convincing the FDA that they do so that they can run better (and potentially cheaper) trials.

I'm sorry, but comments like this are just nonsensical. Do you think that all of a sudden the entire NIH budget, all $30+ billion a year of it, is just going to be spent on NCATS? HEY, WE GET IT, YOU DON'T LIKE THE IDEA OF NCATS! Guess what? NIH still spends lots of money on lots of other research too.

In fact, less than five seconds searching RePORTER for 'biomarkers' would reveal 3499 funded projects, 181 clinical trials, and 280 patents. Yeah, clearly, NIH is doing nothing with regard to biomarkers...

http://projectreporter.nih.gov/reporter_SearchResults.cfm?icde=8345058

Permalink to Comment

21. cliffintokyo on June 5, 2011 9:12 PM writes...

Surely the key question is how useful BM are to physicians in helping them to select an effective treatment for a patient.
It seems many of the new generation of BM are not reliable enough to be validated.
I also would hope the folks at NIH recognize that this is an area where they can probably make a valuable contribution to enhancing new product development, as commented on above.
I imagine that the Regulatory Authorities are paying close attention to the lit on this *fashionable* research trend, and the ominous signs of the usual overhype/overplay.

Permalink to Comment

22. Hap on June 6, 2011 9:14 AM writes...

#20: If a billion out of thirty isn't much, then send me 3.3% of your salary. I'm sure I can find something useful to do with it.

Permalink to Comment

23. Jonathan on June 6, 2011 9:50 AM writes...

NCATS' budget isn't going to be $1 billion. It's like you're railing against some imaginary version of NIH that bears a passing resemblance to the real thing.

Permalink to Comment

24. Hap on June 6, 2011 1:45 PM writes...

Let's try a less mean version of my previous comment.

From the horse's mouth: http://nihrecord.od.nih.gov/newsletters/2011/03_18_2011/story1.htm

$700M seems like a pretty good chunk of $1B. Want to go for 2.3%?

Permalink to Comment

25. Jonathan on June 8, 2011 9:30 AM writes...

Most of that $700 million is existing programs moving from NCRR to NCATS, the bulk of it ($480 million) being CTSAs, which are hardly duplicating work being done by industry. TRND gets a bump, which one would think was a good thing, since TRND's work has been praised by Derek on this blog in the past.

http://blogs.nature.com/news/2011/06/nih_provides_capitol_hill_with_1.html

Permalink to Comment

POST A COMMENT




Remember Me?



EMAIL THIS ENTRY TO A FRIEND

Email this entry to:

Your email address:

Message (optional):




RELATED ENTRIES
How Not to Do It: NMR Magnets
Allergan Escapes Valeant
Vytorin Actually Works
Fatalities at DuPont
The New York TImes on Drug Discovery
How Are Things at Princeton?
Phage-Derived Catalysts
Our Most Snorted-At Papers This Month. . .