About this Author
DBL%20Hendrix%20small.png College chemistry, 1983

Derek Lowe The 2002 Model

Dbl%20new%20portrait%20B%26W.png After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases. To contact Derek email him directly: Twitter: Dereklowe

Chemistry and Drug Data: Drugbank
Chempedia Lab
Synthetic Pages
Organic Chemistry Portal
Not Voodoo

Chemistry and Pharma Blogs:
Org Prep Daily
The Haystack
A New Merck, Reviewed
Liberal Arts Chemistry
Electron Pusher
All Things Metathesis
C&E News Blogs
Chemiotics II
Chemical Space
Noel O'Blog
In Vivo Blog
Terra Sigilatta
BBSRC/Douglas Kell
Realizations in Biostatistics
ChemSpider Blog
Organic Chem - Education & Industry
Pharma Strategy Blog
No Name No Slogan
Practical Fragments
The Curious Wavefunction
Natural Product Man
Fragment Literature
Chemistry World Blog
Synthetic Nature
Chemistry Blog
Synthesizing Ideas
Eye on FDA
Chemical Forums
Symyx Blog
Sceptical Chymist
Lamentations on Chemistry
Computational Organic Chemistry
Mining Drugs
Henry Rzepa

Science Blogs and News:
Bad Science
The Loom
Uncertain Principles
Fierce Biotech
Blogs for Industry
Omics! Omics!
Young Female Scientist
Notional Slurry
Nobel Intent
SciTech Daily
Science Blog
Gene Expression (I)
Gene Expression (II)
Adventures in Ethics and Science
Transterrestrial Musings
Slashdot Science
Cosmic Variance
Biology News Net

Medical Blogs
DB's Medical Rants
Science-Based Medicine
Respectful Insolence
Diabetes Mine

Economics and Business
Marginal Revolution
The Volokh Conspiracy
Knowledge Problem

Politics / Current Events
Virginia Postrel
Belmont Club
Mickey Kaus

Belles Lettres
Uncouth Reflections
Arts and Letters Daily
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline

In the Pipeline

« Enzyme Humility | Main | Three Things You Need »

October 15, 2007

Checking The Numbers on the Alzheimer's Test

Email This Entry

Posted by Derek

The news of a possible diagnostic test for Alzheimer’s disease is very interesting, although there’s always room to wonder about the utility of a diagnosis of a disease for which there is little effective therapy. The sample size for this study is smaller than I’d like to see, but the protein markers that they’re finding seem pretty plausible, and I’m sure that many of them will turn out to have some association with the disease.

But let’s run some numbers. The test was 91% accurate when run on stored blood samples of people who were later checked for development of Alzheimer’s, which compared to the existing techniques is pretty good. Is it good enough for a diagnostic test, though? We’ll concentrate on the younger elderly, who would be most in the market for this test.The NIH estimates that about 5% of people from 65 to 74 have AD. According to the Census Bureau (pdf), we had 17.3 million people between those ages in 2000, and that’s expected to grow to almost 38 million in 2030. Let’s call it 20 million as a nice round number.

What if all 20 million had been tested with this new method? We’ll break that down into the two groups – the 1 million who are really going to get the disease and the 19 million who aren’t. When that latter group gets their results back, 17,290,000 people are going to be told, correctly, that they don’t seem to be on track to get Alzheimer’s. Unfortunately, because of that 91% accuracy rate, 1,710,000 people are going to be told, incorrectly, that they are. You can guess what this will do for their peace of mind. Note, also, that almost twice as many people have just been wrongly told that they’re getting Alzheimer’s than the total number of people who really will.

Meanwhile, the million people who really are in trouble are opening their envelopes, and 910,000 of them are getting the bad news. But 90,000 of them are being told, incorrectly, that they’re in good shape, and are in for a cruel time of it in the coming years.

The people who got the hard news are likely to want to know if that’s real or not, and many of them will take the test again just to be sure. But that’s not going to help; in fact, it’ll confuse things even more. If that whole cohort of 1.7 million people who were wrongly diagnosed as being at risk get re-tested, about 1.556 million of them will get a clean test this time. Now they have a dilemma – they’ve got one up and one down, and which one do you believe? Meanwhile, nearly 154,000 of them will get a second wrong diagnosis, and will be more sure than ever that they’re on the list for Alzheimer’s.

Meanwhile, if that list of 910,000 people who were correctly diagnosed as being at risk get re-tested, 828 thousand of them will hear the bad news again and will (correctly) assume that they’re in trouble. But we’ve just added to the mixed-diagnosis crowd, because almost 82,000 people will be incorrectly given a clean result and won’t know what to believe.

I’ll assume that the people who got the clean test the first time will not be motivated to check again. So after two rounds of testing, we have 17.3 million people who’ve been correctly given a clean ticket, and 828,000 who’ve been correctly been given the red flag. But we also have 154,000 people who aren’t going to get the disease but have been told twice that they will, 90,000 people who are going to get it but have been told that they aren’t, and over 1.6 million people who have been through a blender and don’t know anything more than when they started.

Sad but true: 91% is just not good enough for a diagnostic test. And getting back to that key point in the first paragraph, would 100% be enough for a disease that we can't do anything about? Wait for an effective therapy, is my advice, and for a better test.

Update: See the comments for more, because there's more to it than this. For one thing, are the false positive and false negative rates for this test the same? (That'll naturally make a big difference). And how about differential diagnosis, using other tests to rule out similar conditions? On the should-you-know question, what about the financial and estate planning implications of a positive test - shouldn't those be worth something? (And there's another topic that no one's brought up yet: suicide, which you'd have to think would be statistically noticeable. . .)

Comments (21) + TrackBacks (0) | Category: Alzheimer's Disease | Biological News


1. Michael on October 15, 2007 9:34 PM writes...

This assumes the existing methods are 100% accurate which is far from a given, no?

Could it not be fairly easily be the case that the new test is 99% accurate, but the existing 'by hand' diagnosis is only 90% accurate leading (some of) to the observed discrepency?

Permalink to Comment

2. John Johnson on October 15, 2007 9:54 PM writes...

Gotta push back on this one. Having a test like this (as we do for Huntington's which, unlike this one, is about as definitive as you can get) enables clinical trials for the prevention/early mitigation of AD.

Granted, hopefully there will be another diagnostic test which can be used in combination with this one to provide more specific results, much like they've been doing for years in HIV. A good AD testing system will, of course, improve the quality of early intervention trials.

On the flip side, there is the potential for suicide for people who test positive, which would be doubly tragic for the false positives. We definitely have to approach diagnostics for neurodegeneratives very carefully.

Overall, I think that even at this stage of the game advancing the diagnostics will be good for research, and hopefully applied research in the next few years.

Permalink to Comment

3. Chris on October 15, 2007 10:23 PM writes...

You are A) erroneously assuming that specificity and sensitivity are the same, and are what they mean by "accuracy," and B) probably incorrectly assuming that the problems with the test are probabilistic, rather than systematic (repeated tests will get the same result).

If they were probabilistic, and the sensitivity=specificity=91%, it would be very, very easy to simply have everyone take ten or twenty tests, resulting the chance of an incorrect diagnosis being less than the chance that the person will get sideswiped by a tractor-trailer on the way to the testing center.

And as a person above noted, the fact is, psychology is still a very soft science. There is a reason that there is no consensus on whether ADD is one, two, seven, a thousand, or 0 diseases (some say it's merely a personality subtype). There is a reason we call it "the autistic spectrum," rather than "autism." There is a reason that prosopagnosia, the utter inability to recognise other people's faces, was only acknowledged as a common disorder, rather than a unique condition, with the advent of the search engine.

It's entirely possible that the error rate of the human examiner in finding signs of a distinct physical disorder of varying intensity and progression is greater than the test.

Permalink to Comment

4. PG on October 16, 2007 4:07 AM writes...

I would argue that any *accurate* diagnosis is useful even if there is currently no effective therapy. In the case of diseases like Alzheimer, which has such a long-term burder (physical, emotional, and financial) on spouses, families, and friends, a person diagnosed with a very high risk of getting it may be able to make choices (like saving money for future care) that he/she might not have made otherwise.

Permalink to Comment

5. Anonymous on October 16, 2007 7:13 AM writes...

91% isn't nearly good enough for a front line, mass population, diagnostic. But, if it's given when there are other symptoms presenting, then it's OK at best. It is also very important to distinguish between false positives and false negatives. I have 1st hand experience with this: cancer screen is 90% accurate, with only false positives (false negatives are very rare). You can be clean, but 1 out of 10 times it'll turn up positive. Then you just get a CAT scan and redo the test. Pain in the butt? Expensive? Yes and yes. Peace of mind knowing you're clean? Priceless.

In the Alz screen (link?), are the false positives and negatives evenly distributed, or is one more prevalent? This is an important distinction. In your example of 20M people, if it only gives false positives, the test is 100% accurate for the 1M that will get Alz; there are no 91,000 being told they are OK. The 19M clean population is still getting hosed.

Permalink to Comment

6. A-non-y-mous on October 16, 2007 7:33 AM writes...

Sorry, I didn't give my "name" to my comment above.

Permalink to Comment

7. John Spevacek on October 16, 2007 7:59 AM writes...

Repeating the test may give the same 91/9 split to the same people no matter how many times they take it if there is an uncontrolled variable floating around.

It reminds me of drug testing athletes. The initial urine or blood draw is split into an A and B. If A is positive, then the B is also tested by the same lab using the same equipment. Is it any wonder that the B test result is always the same as the A test result?

Permalink to Comment

8. HelicalZz on October 16, 2007 8:44 AM writes...

Is 91% good enough. Depends on the test and what it is used for. If it is simply a cheap screening then yes it is probably good enough.

For example, lets say I have an injectable reagent which combined with a brain CT scan is 99% accurate (and these are in development for Alzheimer's), but it costs $2K. I also have a protein chip or test similar to the one discussed that costs $150, but is only 91% accurate. Then the first test should certainly be used as a justification for the second more expensive test (simple economics).

So in general, 91% isn't always good enough, but sometimes it is.


Permalink to Comment

9. qetzal on October 16, 2007 9:00 AM writes...

Some numbers from the paper:

The 91% comes from testing a group of 47 people who were first diagnosed as having mild cognitive impairment (MCI). They gave blood samples at the start, and then were followed up over 2-6 years. During that time, 22 developed "confirmed" Alzheimer's (AD), 8 developed other dementias (OD), and 17 showed no progression.

When the test was run on the time 0 blood samples, it correctly predicted 20 of the 22 patients who developed AD (20/22 = 91% true positives; 2/22 = 9% false negatives).

None of the 8 people who developed OD tested positive at time 0. However, 7 of 17 people who did not develop AD or OD tested positive. If you combine these two groups to get all people who did not develop AD, the false positive rate was 7/25, or 28%.

That's pretty high.

It's also important to consider the tested population. These weren't random older people. These were people who were already diagnosed with MCI. They had a much higher risk of AD than the general population. 43% (20/47) developed AD within 6 years! And even then, the test had a very high false positive rate.

Hard to say what might happen if a healthy population was tested. Maybe the false positive rate would be much lower. For example, some of the above false positives might not be false. Follow-up was only ~ 6 years, so maybe some or all of those 7 will convert to AD later.

OTOH, with a 28% false positive rate even among people who are like to develop AD, I wouldn't be surprised to see a similarly high rate among healthy subjects.

Don't get me wrong. I think these results are extremely promising from a research perspective. But it seems clear that they still fall short of having general prognostic value.

P.S. I agree with John Spevacek. I doubt the false positives and negatives represent random errors. More likely, they are correctly measuring whether the blood markers do or do not fit the 'AD profile.' Some people will almost certainly have that profile, but never actually develop AD. Others will not have the profile, but will develop AD. You could repeat the same test on them as much as you like, and always get the same result, without changing your true ability to predict disease.

Permalink to Comment

10. qetzal on October 16, 2007 9:32 AM writes...

The statistics of diagnostic testing in generally healthy populations can be a bit counterintuitive. Let’s take Derek’s number of 20 million elderly with a 5% chance of developing AD. Let’s further assume we have a test that predicts AD with a 10% chance of false positives and a 10% chance of false negatives.

We test all 20 million. Of the 1 million that will go on to develop AD, 900K will test positive (true positives) and 100K will test negative (false negatives). Of the 19 million that will not develop AD, 17.1 million will test negative (true negatives) and 1.9 million will test positive (false positives).

Now imagine you’re one of those 20 million. Before you take the test, you know you have a 5% chance of developing AD. You take the test and get a positive result. What does that mean? Either you’re one of the 900K true positives, or one of the 1.9 million false positives. So now your chance of developing AD is 32%. Your odds of developing AD went from 1:20 to 1:3.

In other words, even though the test gives the correct result 90% of the time, we still can’t say which people are likely (> 50% chance) to develop AD.

Permalink to Comment

11. Anonymous on October 16, 2007 11:50 AM writes...

How can you run a clinical trial on a potential Alzheimer's treatment without knowing who has it and who doesn't? So yes, a diagnostic test would be a great thing to have even without a cure/treatment currently, and I stress currently, available.

Permalink to Comment

12. Klug on October 16, 2007 1:48 PM writes...

Could someone post numbers on what a gold standard medical test is? I would imagine that it's a lot higher than 91%, but I don't know.

Permalink to Comment

13. Chris P on October 16, 2007 5:35 PM writes...

If the autopsy data shows the signs of Alzheimer's disease (plaques and tangles) but the clinical signs were not manifested pre-mortem, then did the person have Alzheimer's disease or were they in a pro-dromal state that we call normal? Those of us working in this area do not have a flawless biomarker (like cholesterol or blood pressure) to measure to see if someone is at risk for an event (heart attack, stroke, death). Some have pointed to this problem when trying to explain their trial results (a mixed population). I believe clinical diagnosis rate is only in the high 80s for sensitivity and specificity compared to the post-mortem data. This is one reason why none of the guidelines ask for a dementia screen.

In the Facing Dementia Survey 35% of physicians believed that specialists have a hard time diagnosing AD.

Permalink to Comment

14. Anonymous on October 17, 2007 1:04 PM writes...

I am not sure what the value of this test would be, and why any person would take this test. The social and ethical issues with this type of test are huge. The lack of value (prognostic, actionable value) for a physician would prevent the reimbursement by the insurers, so it would be an out-of-pocket test. Who in their right mind would plunk down a bunch of money to get a piece of paper that says, "Hey, you are going to get AD! Maybe. Or Maybe not! Either way, do not tell your insurance company about this test or they will exclude AD treatments as it was a pre-existing or pre-ordained condition. That will be $5000 please." On the other hand, if the information from these proteins can be refined, there may be treatments in the future. Have they looked at the RNA? I am not sure that pure protein screening is the answer, but a lot of smart people are looking in that direction.

Permalink to Comment

15. Dr. Hetzel on October 18, 2007 12:42 PM writes...

Yes, the 'comments' get to the meat, and thus are more useful than the blog itself.

'Update' should be entitled, 'Errata'.

Permalink to Comment

16. WonderDrug on October 18, 2007 5:12 PM writes...

I don't see what the big deal is here. There are risks associated with every test. Notifying the patient and making sure he understands the probabilities is important. There is no reason to discount the test which is still better than no test at all.

Permalink to Comment

17. Derek Lowe on October 18, 2007 8:51 PM writes...

Dr. H, the entry barrier to starting a blog is as low as can be. Start one, and I'll link to it.

Permalink to Comment

18. leslie graham on October 26, 2007 6:28 PM writes...

is there an actual connection between proton pump inhibiters and alzheimers?

Permalink to Comment

19. leslie graham on October 26, 2007 6:30 PM writes...

is there an actual connection between proton pump inhibiters and alzheimers? i have never posted before so why does this system say that i have posted too many times?

Permalink to Comment

20. David VanLandingham on October 28, 2007 5:56 PM writes...

There are excellent reasons for an early diagnosis of Alzheimer’s. Current drugs slow the progress, drugs in Phase III trials have the potential to stop the progress and perhaps even reverse some of the damage. An early diagnosis allows a person to take meds that help them retain their abilities until the drugs in Phase III trials come onto the market.

Additionally, an early diagnosis allows the patient to make financial, medical and legal determinations before they are deemed incompetent.

Permalink to Comment

21. rc on October 11, 2012 9:23 AM writes...

Your hypothesis is severely flawed. You do not test all 20 million people, of course that would yield a large number of false positives no matter how accurate the test.

You only test the patients where there is a reason to suspect they have the disease. The rate of false positives drops dramatically under this paradigm.

You know better, not sure why you proposed testing an entire population to illustrate your point. You could make even the best test look bad under those circumstances.

Permalink to Comment


Remember Me?


Email this entry to:

Your email address:

Message (optional):

The Last Post
The GSK Layoffs Continue, By Proxy
The Move is Nigh
Another Alzheimer's IPO
Cutbacks at C&E News
Sanofi Pays to Get Back Into Oncology
An Irresponsible Statement About Curing Cancer
Oliver Sacks on Turning Back to Chemistry