Corante

About this Author
DBL%20Hendrix%20small.png College chemistry, 1983

Derek Lowe The 2002 Model

Dbl%20new%20portrait%20B%26W.png After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases. To contact Derek email him directly: derekb.lowe@gmail.com Twitter: Dereklowe

Chemistry and Drug Data: Drugbank
Emolecules
ChemSpider
Chempedia Lab
Synthetic Pages
Organic Chemistry Portal
PubChem
Not Voodoo
DailyMed
Druglib
Clinicaltrials.gov

Chemistry and Pharma Blogs:
Org Prep Daily
The Haystack
Kilomentor
A New Merck, Reviewed
Liberal Arts Chemistry
Electron Pusher
All Things Metathesis
C&E News Blogs
Chemiotics II
Chemical Space
Noel O'Blog
In Vivo Blog
Terra Sigilatta
BBSRC/Douglas Kell
ChemBark
Realizations in Biostatistics
Chemjobber
Pharmalot
ChemSpider Blog
Pharmagossip
Med-Chemist
Organic Chem - Education & Industry
Pharma Strategy Blog
No Name No Slogan
Practical Fragments
SimBioSys
The Curious Wavefunction
Natural Product Man
Fragment Literature
Chemistry World Blog
Synthetic Nature
Chemistry Blog
Synthesizing Ideas
Business|Bytes|Genes|Molecules
Eye on FDA
Chemical Forums
Depth-First
Symyx Blog
Sceptical Chymist
Lamentations on Chemistry
Computational Organic Chemistry
Mining Drugs
Henry Rzepa


Science Blogs and News:
Bad Science
The Loom
Uncertain Principles
Fierce Biotech
Blogs for Industry
Omics! Omics!
Young Female Scientist
Notional Slurry
Nobel Intent
SciTech Daily
Science Blog
FuturePundit
Aetiology
Gene Expression (I)
Gene Expression (II)
Sciencebase
Pharyngula
Adventures in Ethics and Science
Transterrestrial Musings
Slashdot Science
Cosmic Variance
Biology News Net


Medical Blogs
DB's Medical Rants
Science-Based Medicine
GruntDoc
Respectful Insolence
Diabetes Mine


Economics and Business
Marginal Revolution
The Volokh Conspiracy
Knowledge Problem


Politics / Current Events
Virginia Postrel
Instapundit
Belmont Club
Mickey Kaus


Belles Lettres
Uncouth Reflections
Arts and Letters Daily
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline

In the Pipeline

« AbbVie and Shire, Quietly | Main | Outsourced Assays, Now a Cause For Wonder? »

July 8, 2014

An Alzheimer's Blood Test? Not So Fast.

Email This Entry

Posted by Derek

There all all sorts of headlines today about how there's going to be a simple blood test for Alzheimer's soon. Don't believe them.

This all comes from a recent publication in the journal Alzheimer's and Dementia, from a team at King's College (London) and the company Proteome Sciences. It's a perfectly good paper, and it does what you'd think: they quantified a set of proteins in a cohort of potential Alzheimer's patients and checked to see if any of them were associated with progression of the disease. From 26 initial protein candidates (all of them previously implicated in Alzheimer's), they found that a panel of ten seemed to give a prediction that was about 87% accurate.

That figure was enough for a lot of major news outlets, who have run with headlines like "Blood test breakthrough" and "Blood test can predict Alzheimer's". Better ones said something more like "Closer to blood test" or "Progress towards blood test", but that's not so exciting and clickable, is it? This paper may well represent progress towards a blood test, but as its own authors, to their credit, are at pains to say, a lot more work needs to be done. 87%, for starters, is interesting, but not as good as it needs to be - that's still a lot of false negatives, and who knows how many false positives.

That all depends on what the rate of Alzheimer's is in the population you're screening. As Andy Extance pointed out on Twitter, these sorts of calculations are misunderstood by almost everyone, even by people who should know better. A 90 per cent accurate test on a general population whose Alzheimer's incidence rate is 1% would, in fact, be wrong 92% of the time. Here's a more detailed writeup I did in 2007, spurred by reports of a similar Alzheimer's diagnostic back then. And if you have a vague feeling that you heard about all these issue (and another blood test) just a few months ago, you're right.

Even after that statistical problem, things are not as simple as the headlines would have you believe. This new work is a multivariate model, because a number of factors were found to affect the levels of these proteins. The age and gender of the patient were two real covariants, as you'd expect, but the duration of plasma storage before testing also had an effect, as did, apparently, the center where the collection was done. That does not sound like a test that's ready to be rolled out to every doctor's office (which is again what the authors have been saying themselves). There were also different groups of proteins that could be used for a prediction model using the set of Mild Cognitive Impairment (MCI) patients, versus the ones that already appeared to show real Alzheimer's signs, which also tells you that this is not a simple turn-the-dial-on-the-disease setup. Interestingly, they also looked at whether adding brain imaging data (such as hippocampus volume) helped the prediction model. This, though, either had no real effect on the prediction accuracy, or even reduced it somewhat.

So the thing to do here is to run this on larger patient cohorts to get a more real-world idea of what the false negative and false positive rates are, which is the sort of obvious suggestion that is appearing in about the sixth or seventh paragraph of the popular press writeups. This is just what the authors are planning, naturally - they're not the ones who wrote the newspaper stories, after all. This same collaboration has been working on this problem for years now, I should add, and they've had ample opportunity to see their hopes not quite pan out. Here, for example, is a prediction of an Alzheimer's blood test entering the clinic in "12 to 18 months", from . . .well, 2009.

Update: here's a critique of the statistical approaches used in this paper - are there more problems with it than were first apparent?

Comments (30) + TrackBacks (0) | Category: Alzheimer's Disease | Analytical Chemistry | Biological News


COMMENTS

1. Biotechtranslated on July 8, 2014 11:31 AM writes...

Even if such a test were accurate and ready for the market, the big question is "so what?" There are no therapies to treat Alzheimer's disease (beyond symptomatic treatments). What value, beyond the research setting, would such a test provide?

Permalink to Comment

2. SM on July 8, 2014 11:45 AM writes...

@ #1: I think the potential of a test for pre-Alz would be huge. Since reversing the effects of Alzheimers is probably insanely difficult if not impossible the best strategy would be to prevent it. A test to determine if Alzheimers is going to be an issue sets a precedence for a realistic clinical trial scenario for a therapeutic that might prevent the onset. Without a way of knowing ahead of time who will develop Alz testing therapeutics is extremely hard.

Permalink to Comment

3. Anonymous on July 8, 2014 12:33 PM writes...

I have a 100% accurate test that tells you will die, eventually. So what? This spreading of false hope is worse than useless, and not worth the paper its written on.

Permalink to Comment

4. respisci on July 8, 2014 12:42 PM writes...

In the news articles I have read about this there are suggestions to use these blood markers as surrogates in clinical trials. I foresee Phase 2 studies showing effects on blood marker levels and then failures in Phase 3 as there is no effect on dementia.

Permalink to Comment

5. Miramon on July 8, 2014 2:12 PM writes...

This is an excellent blog post that deserves to be made into an editorial for some newspaper like the NY Times.

Just that observation on the incidence of the disease affecting the percentage of false positives reported in the population is enough to justify the piece by itself, though it might require a few more words of explanation for the great unwashed to get the idea.

Permalink to Comment

6. David Grainger on July 8, 2014 3:13 PM writes...

If only it was a "perfectly good paper". The multivariate statistical treatment is deeply flawed on several levels. The univariate analyses are mostly not corrected for multiple comparisons; the multivariate models are mostly not tested for generalisability; and the headline ROC models are not tested for significance or for independence from previously known risk-factors.

All the concerns about how to properly qualify a diagnostic test for clinic use made here are perfectly valid (unlike the paper itself). But before you even worry about what an "87% accurate" test might mean, you need to have a test that is robustly 87% accurate. This one certainly isn't that!

My full critique of the statistical analysis in the paper is here: http://www.tcpinnovations.com/drugbaron/twenty-six-of-the-best-candidate-protein-biomarkers-have-no-ability-to-predict-conversion-to-dementia-from-prodromal-disease/

Permalink to Comment

7. Any Mouse on July 8, 2014 3:13 PM writes...

"A 90 per cent accurate test on a general population whose Alzheimer's incidence rate is 1% would, in fact, be wrong 92% of the time." No, it is still accurate 90% of the time.

You need the all important qualifier "given a positive result" ...

Permalink to Comment

8. exGlaxoid on July 8, 2014 3:31 PM writes...

As someone who has a lot of AZ in the family, including several parents, I do think that there is a place for knowing with some likelihood that someone will get AZ. That allows you to prepare things for the eventual move from a house to a care facility. It gives the family time to prepare a power of attorney, add children to bank accounts, make sure that insurance premiums are being paid and bills are being watched out for.

It also does allow you more chance to keep active, mentally and physically, which appears to help slow the progression (as much as anything else can, at least). The person with AZ may not benefit from knowing, but their care can be planned for better, and their children can be prepared for the eventuality, at least. They might also be able to get the parent to wear an emergency alarm or GPS tracker.

I can tell you that even with proper preparation (POA, knowing where the documents and accounts are located, third party notification on bills, etc) it is not easy to care for a parent with dementia, but without them, it is a complete nightmare. And this allows some time to prepare a plan, as moving into some sort of senior living is much easier when the affected person can help pick the location, decide on the details, and help move.

This is much better than one day finding your parents are lost, getting a call from the police that they are miles away and don't know who they are, have been fleeced for all of their saving, or other equally bad calls. All of those have happened to people I know when a parent suddenly became senile, although often there were hints that went unnoticed.

And if any of the tests can help in clinical trials, even better, although I agree that they are not perfect. But for people with AZ, any hope for a cure is better than none.

Permalink to Comment

9. Eskimo on July 8, 2014 3:45 PM writes...

Isn't the basic idea of these studies to get these biomarker panels used in clinical trials as a gatekeeper for enrollment -- say for a BACE inhibitor or something?
A clinical cardiology researcher I know told me that that industry is less interested in these types of risk-predicting-biomarker-panels after the Supreme Court's Myriad decision. Derek, you and your commenters could probably provide some perspective on that.

Permalink to Comment

10. Eskimo on July 8, 2014 3:46 PM writes...

Isn't the basic idea of these studies to get these biomarker panels used in clinical trials as a gatekeeper for enrollment -- say for a BACE inhibitor or something?
A clinical cardiology researcher I know told me that that industry is less interested in these types of risk-predicting-biomarker-panels after the Supreme Court's Myriad decision. Derek, you and your commenters could probably provide some perspective on that.

Permalink to Comment

11. Avi on July 8, 2014 3:47 PM writes...

@6 Perhaps the cleanest way to say it is "If the actual disease incidence rate is 1%, then a test with a 90% (true positive) accuracy rate is going to misdiagnose (supply a false positive) almost 91.7% of the time. A classic example of Bayes law.

Permalink to Comment

12. luysii on July 8, 2014 3:55 PM writes...

Bravo #7 ExGlaxoid -- that's the way Alzheimer's is, and I saw this many times as a neurologist. Good to have the reality of the disorder in the comments, painful as it must have been to write your note.

As far as the statistics of a false positive test and the likelihood that it actually meant something goes, I had to explain this at least once a week to a frightened patient.

Back then we could get a metabolic profile of 20 or so blood tests (electrolytes, kidney liver function etc. etc.). The bounds of normal excluded the top 2.5% and the bottom 2.5%. So assuming you were in fact normal, the chance of any individual test falling in the normal range was ... 95%. However, the chance of all 20 of them doing so was --- .95^20 = 35.8% which meant that abnormal chem20s were twice as common as normal ones.

Permalink to Comment

13. metaphysician on July 8, 2014 4:15 PM writes...

#10-

So, if I'm understanding the situation right. . . the problem is that, there are far, far more people who actually don't have the disease, than people who actually do. So, statistically speaking, a positive test result is far more likely to be a false positive than true, because 10% of 99% is a much larger number than 90% of 1%.
Conversely, a negative test result is also effected, but since the numbers are in favor of not having the disease, this doesn't matter.

If I understand the situation properly, the test would need to be at least two orders of magnitude more accurate to be worthwhile ( one order would make a positive result a coin flip, which is probably not good enough ). This is assuming there's actually a 1% Alzheimer's rate, mind.

Permalink to Comment

14. Avi on July 8, 2014 4:26 PM writes...

@12, In a nutshell, yup. Only 0.9% of all people simultaneously have positive tests and the disease, yet 10.8% of all people are going to get positive results (those 0.9% plus the 9.9% of healthy people who fall into the 10% false positive rate). This means that only 0.9 / 10.8, or about 8.3% of the positives are real, so the remaining 91.7% or so are false.

Forgiving the statistical intrusion into a chemistry blog, using the terminology of Bayes' theorem, if + means a positive test and D means the person is diseased, then we know that P(+|D), or the probability that a person has the disease given the test comes back positive, is 90%, but P(+), the overall incidence of disease, is only 1%. We want to know P(D|+), or the probability that there is presence of the disease given a positive test. Bayes' theorem allows us to calculate that as P(D|+) = P(+|D)*P(D) / P(+).

Permalink to Comment

15. Avi on July 8, 2014 4:39 PM writes...

Quick correction. The second paragraph should read :if + means a positive test and D means the person is diseased, then we know that P(+|D), or the probability that a person will test positive given presence of the disease, is 90%…"

Permalink to Comment

16. Biotechtranslated on July 8, 2014 6:51 PM writes...

@#7

Your points are well taken. I read this as a test of progression, that is, someone starts to show prodromal signs and gets tested to confirm AD. I think that has relatively limited value outside of a research setting.

Many of the benefits you outlined are around very early detection. Is there value in telling a 20 or 30 yr old that they are likely to get AD? Sure! It's the same as being tested for a cancer gene.

Permalink to Comment

17. Third Wind on July 8, 2014 10:32 PM writes...

All of this is fine and very dandy until Wall Street management barges in and demands maximization of numbercules/unit time.
Consider what really happened to me a while ago: I asked a Key-Opinion-Maker about what HIs Excellency thought of a new Japanese diagnostic method : "it takes 1/2 hour vs. 10 minutes to get the number so insurance will pay less or not pay". How Japanese life expectancy is so high could be another post and just possibly related, but, staying on this particular "blog meeting", could we capture the purpose before conclusions/next actions are drawn? "Reimbursement" "Quarterly profit" "Car Elevator" "Or What"

Permalink to Comment

18. matt on July 8, 2014 11:00 PM writes...

@Biotechtranslated #1, #16: I couldn't disagree more strenuously. Perhaps it's a personal choice, and you'd prefer ignorance. Not me.

Maybe this is the Lou Gehrig Test: do you think Gehrig's doctor should have lied to him about his situation, or kept him in the dark? I don't think that's appropriate, unless the patient makes it clear that's what they want.

I'd want to know: how bad is my situation now, relative to the progression of the disease? How fast is it progressing? These would be indicators of how much time do I have, can I trust my driving, my personal banking, my ability to take care of myself, or is it time to ask for help and give up control? If I'm going to ask to move in with somebody, or ask somebody to move in to care for me, or sell my house and move to an assisted living community, is it time or how quickly does that have to happen? If I have some other medical condition, for example cancer, how worthwhile is it to treat the cancer, or would I really prefer the body to give out before the mind?

It may be a long time before we approach any accuracy, and of course at best we still just get statistical information, but we certainly aren't getting anywhere if we fail to make the effort.

From a clinical point of view, it would be an enormous benefit to drug trials to have more accurate indications of even the presence or absence of AD, much less (dare we even hope) for accurate indicators of progression or changing risk. In some ways, because the cost of the drug trials is dependent on showing a benefit, and showing a benefit is nearly impossible in our current "cloud of unknowing" about the state of AD, the hope for a cure likely DEPENDS on more accurate testing.

Not just that, but it's likely that different treatments will be useful at different phases of the disease. Not only will it be hard to find such treatments if we can't discriminate between phases or progression of disease, but we wouldn't be able to apply them, either.

The logic, therefore, that such work is useless infuriates me with its shortsightedness. It may have limited value for someone very far along with AD (they don't need a test), but for someone just being diagnosed--besides the value mentioned above for taking the necessary preparations--it's crucial to make things easier for a cure to be found. And that seems to me to be the most valuable benefit of all.

Permalink to Comment

19. Off Topic on July 9, 2014 12:38 AM writes...

Every chemist worth his salt has a story or two about cleaning out an old lab and finding a bottle of something scary. You know it's a good one if the bomb squad gets called in, and banter tends to go back and forth about who had the 'worst' vial of doom.

The NIH just one-upped everyone by discovering *smallpox* in an old lab.

http://www.cdc.gov/media/releases/2014/s0708-NIH.html

Permalink to Comment

20. Anchor on July 9, 2014 5:55 AM writes...

Simonian! O Brother, Where Art Thou?

Permalink to Comment

21. Anonymous on July 9, 2014 8:33 AM writes...

Avi,

But the test is proposed as a way of predicting which people with mild cognitive impairment will go on to AZ, not a way to look 30 years into the future. So the fact that only 1% of the population has AZ at any one time does not matter, what matters is what percentage of people with MCI will get AZ. So that analysis is flawed.

What matters is what percentage of the people with mild cognitive impairment will get so much worse that they need special care. About 25% of the MCI patients developed AZ in one year, so that means that if the test can predict those 51 out of 220 with MCI, that would be a much higher proportion that 1%.

I agree that the work is early and not perfect (I'm not related to the study in any way) but if there was even a reasonable predictor of the progression of the disease, that could be very useful to patients and their families. I don't think anyone can predict AZ for 10 to 30 years out very accurately, no better than cancer or heart disease (they are likely much easier to predict). But knowing that your mother has a 91% +- 10% chance of worsening within a year might be very helpful, and in the false positive case that they stay well, most people would be happy. I would be much more concerned with false negatives, as if you think someone is not going to get sick and they do, that is often more of a problem.

But no test can ever avoid that risk, as you could get hit by a bus or struck by lightening at nearly anytime.

Permalink to Comment

22. John-john on July 9, 2014 9:06 AM writes...

There are a staggering number of red flags in that paper- they used SPSS, no Benferroni corrections for multiple testing, and the methods sections is garbled and stumbling. Moreover, there are no CIs for the entire ROC analysis (which is a trivial option) so that entire part of the paper is total and complete dreck. It's exceedingly likely the ROC curves in figure 3 all totally overlap- and why exactly did they not present the full odds ratios with 95% CIs from the logistic regression??? !!!

Permalink to Comment

24. Anonymous on July 9, 2014 9:29 AM writes...

@22, what's the problem with using SPSS?

Permalink to Comment

25. a. nonymaus on July 9, 2014 9:38 AM writes...

The real test is whether you remember the news about the last test or if you get argumentative about how testing was done back in... :-)

Permalink to Comment

26. the boss on July 9, 2014 10:46 AM writes...

Even if this or some other test works 100%; in the long run, companies will ask for the potential employees' test results. This will be one of their major tools in determining whether they should hire an employee or not. I am not even talking about the insurance companies. That's even worse.

Permalink to Comment

27. Harrison on July 9, 2014 1:17 PM writes...

@8: If there is really that much AD in your family, it probably would be beneficial to be genotyped for APOE for planning purposes. Estimates vary (of course), but two copies of the APOE-4 gene confer about an 80% chance of AD. With one copy of APOE-4, chances of developing AD drop to about 50%, which is why testing isn't routinely done. And of course about 40% of AD patients don't have any copies of APOE-4...

Permalink to Comment

28. John-john on July 9, 2014 4:26 PM writes...

@24: SPSS is not set up to run analyses using scripts- it's possible but difficult, and I'd guess 99% of users analyze via menus. This means the analyses are very error-prone, and have no reproducibility or traceability, unlike essentially every other package (SAS, Stata or R).

Permalink to Comment

29. Trauma on July 23, 2014 8:50 AM writes...

'A 90 per cent accurate test on a general population whose Alzheimer's incidence rate is 1% would, in fact, be wrong 92% of the time'

What daft remarks Derek

You're taking the text out of the context of its likely usage.
The test would be used on patients with MCI problems already. It would be used on older people where the prevalence is higher.
Your prevalence rate is inappropriate and so you're distorting things!

A test kit with the results they've achieved IS a breakthrough
Bear in mind the aim is to identify patients most suitable for a drug approach, to help find efficacy in a practical manner.


You refer to this article as suggesting things havent panned out as the teams hoped
'Here, for example, is a prediction of an Alzheimer's blood test entering the clinic in "12 to 18 months", from . . .well, 2009'

That was for a research assay
It was launched and has been available for clinical work for years.
Take a look in the Millipore catalogues
Again, you're confused

As for the TCP Innovations critique you refer to, this was laughable. The biased author totally distorted the whole thing in his efforts to generate attention

Mainstream press may be criticised for using headline grabbing wording or over-excited views, but let's face it - bloggers are doing exactly the same thing!

Permalink to Comment

30. Trauma on July 23, 2014 8:53 AM writes...

the boss

'Even if this or some other test works 100%; in the long run, companies will ask for the potential employees' test results'

so companies are already asking employees to be tested for cancers etc?

Why can people like you not just stick to honest remarks?

Permalink to Comment

POST A COMMENT




Remember Me?



EMAIL THIS ENTRY TO A FRIEND

Email this entry to:

Your email address:

Message (optional):




RELATED ENTRIES
The Worst Seminar
Conference in Basel
Messed-Up Clinical Studies: A First-Hand Report
Pharma and Ebola
Lilly Steps In for AstraZeneca's Secretase Inhibitor
Update on Alnylam (And the Direction of Things to Come)
There Must Have Been Multiple Chances to Catch This
Weirdly, Tramadol Is Not a Natural Product After All