The news of a possible diagnostic test for Alzheimer’s disease is very interesting, although there’s always room to wonder about the utility of a diagnosis of a disease for which there is little effective therapy. The sample size for this study is smaller than I’d like to see, but the protein markers that they’re finding seem pretty plausible, and I’m sure that many of them will turn out to have some association with the disease.
But let’s run some numbers. The test was 91% accurate when run on stored blood samples of people who were later checked for development of Alzheimer’s, which compared to the existing techniques is pretty good. Is it good enough for a diagnostic test, though? We’ll concentrate on the younger elderly, who would be most in the market for this test.The NIH estimates that about 5% of people from 65 to 74 have AD. According to the Census Bureau (pdf), we had 17.3 million people between those ages in 2000, and that’s expected to grow to almost 38 million in 2030. Let’s call it 20 million as a nice round number.
What if all 20 million had been tested with this new method? We’ll break that down into the two groups – the 1 million who are really going to get the disease and the 19 million who aren’t. When that latter group gets their results back, 17,290,000 people are going to be told, correctly, that they don’t seem to be on track to get Alzheimer’s. Unfortunately, because of that 91% accuracy rate, 1,710,000 people are going to be told, incorrectly, that they are. You can guess what this will do for their peace of mind. Note, also, that almost twice as many people have just been wrongly told that they’re getting Alzheimer’s than the total number of people who really will.
Meanwhile, the million people who really are in trouble are opening their envelopes, and 910,000 of them are getting the bad news. But 90,000 of them are being told, incorrectly, that they’re in good shape, and are in for a cruel time of it in the coming years.
The people who got the hard news are likely to want to know if that’s real or not, and many of them will take the test again just to be sure. But that’s not going to help; in fact, it’ll confuse things even more. If that whole cohort of 1.7 million people who were wrongly diagnosed as being at risk get re-tested, about 1.556 million of them will get a clean test this time. Now they have a dilemma – they’ve got one up and one down, and which one do you believe? Meanwhile, nearly 154,000 of them will get a second wrong diagnosis, and will be more sure than ever that they’re on the list for Alzheimer’s.
Meanwhile, if that list of 910,000 people who were correctly diagnosed as being at risk get re-tested, 828 thousand of them will hear the bad news again and will (correctly) assume that they’re in trouble. But we’ve just added to the mixed-diagnosis crowd, because almost 82,000 people will be incorrectly given a clean result and won’t know what to believe.
I’ll assume that the people who got the clean test the first time will not be motivated to check again. So after two rounds of testing, we have 17.3 million people who’ve been correctly given a clean ticket, and 828,000 who’ve been correctly been given the red flag. But we also have 154,000 people who aren’t going to get the disease but have been told twice that they will, 90,000 people who are going to get it but have been told that they aren’t, and over 1.6 million people who have been through a blender and don’t know anything more than when they started.
Sad but true: 91% is just not good enough for a diagnostic test. And getting back to that key point in the first paragraph, would 100% be enough for a disease that we can't do anything about? Wait for an effective therapy, is my advice, and for a better test.
1. Michael on October 15, 2007 9:34 PM writes...
This assumes the existing methods are 100% accurate which is far from a given, no?
Could it not be fairly easily be the case that the new test is 99% accurate, but the existing 'by hand' diagnosis is only 90% accurate leading (some of) to the observed discrepency?
Permalink to Comment2. John Johnson on October 15, 2007 9:54 PM writes...
Gotta push back on this one. Having a test like this (as we do for Huntington's which, unlike this one, is about as definitive as you can get) enables clinical trials for the prevention/early mitigation of AD.
Granted, hopefully there will be another diagnostic test which can be used in combination with this one to provide more specific results, much like they've been doing for years in HIV. A good AD testing system will, of course, improve the quality of early intervention trials.
On the flip side, there is the potential for suicide for people who test positive, which would be doubly tragic for the false positives. We definitely have to approach diagnostics for neurodegeneratives very carefully.
Overall, I think that even at this stage of the game advancing the diagnostics will be good for research, and hopefully applied research in the next few years.
Permalink to Comment3. Chris on October 15, 2007 10:23 PM writes...
You are A) erroneously assuming that specificity and sensitivity are the same, and are what they mean by "accuracy," and B) probably incorrectly assuming that the problems with the test are probabilistic, rather than systematic (repeated tests will get the same result).
If they were probabilistic, and the sensitivity=specificity=91%, it would be very, very easy to simply have everyone take ten or twenty tests, resulting the chance of an incorrect diagnosis being less than the chance that the person will get sideswiped by a tractor-trailer on the way to the testing center.
And as a person above noted, the fact is, psychology is still a very soft science. There is a reason that there is no consensus on whether ADD is one, two, seven, a thousand, or 0 diseases (some say it's merely a personality subtype). There is a reason we call it "the autistic spectrum," rather than "autism." There is a reason that prosopagnosia, the utter inability to recognise other people's faces, was only acknowledged as a common disorder, rather than a unique condition, with the advent of the search engine.
It's entirely possible that the error rate of the human examiner in finding signs of a distinct physical disorder of varying intensity and progression is greater than the test.
Permalink to Comment4. PG on October 16, 2007 4:07 AM writes...
I would argue that any *accurate* diagnosis is useful even if there is currently no effective therapy. In the case of diseases like Alzheimer, which has such a long-term burder (physical, emotional, and financial) on spouses, families, and friends, a person diagnosed with a very high risk of getting it may be able to make choices (like saving money for future care) that he/she might not have made otherwise.
Permalink to Comment5. Anonymous on October 16, 2007 7:13 AM writes...
91% isn't nearly good enough for a front line, mass population, diagnostic. But, if it's given when there are other symptoms presenting, then it's OK at best. It is also very important to distinguish between false positives and false negatives. I have 1st hand experience with this: cancer screen is 90% accurate, with only false positives (false negatives are very rare). You can be clean, but 1 out of 10 times it'll turn up positive. Then you just get a CAT scan and redo the test. Pain in the butt? Expensive? Yes and yes. Peace of mind knowing you're clean? Priceless.
In the Alz screen (link?), are the false positives and negatives evenly distributed, or is one more prevalent? This is an important distinction. In your example of 20M people, if it only gives false positives, the test is 100% accurate for the 1M that will get Alz; there are no 91,000 being told they are OK. The 19M clean population is still getting hosed.
Permalink to Comment