« Physics, for Dogs and Others |
| Oxford's New Building, One Year Later »
January 18, 2010
Correlations, Lovely Correlations
Anyone looking over large data sets from human studies needs to be constantly on guard. Sinkholes are everywhere, many of them looking (at first glance) like perfectly solid ground on which to build some conclusions. This, to be honest, is one of the real problems with full release of clinical trial data sets: if you're not really up on your statistics, you can convince yourself of some pretty strange stuff.
Even people who are supposed to know what they're doing can bungle things. For instance, you may well have noticed a lot of papers coming out in the last few years correlating neuroimaging studies (such as fMRI) with human behaviors and personality traits. Neuroimaging is a wonderfully wide-open, complex, and important field, and I don't blame people for a minute for pushing it as far as it can go. But just how far is that?
A recent paper (PDF) suggests that the conclusions have run well ahead of the numbers. Recent papers have been reporting impressive correlations between the activation of particular brain regions and associated behaviors and traits. But when you look at the reproducibility of the behavioral measurements themselves, the correlation is 0.8 at best. And the reproducibility of the blood-oxygen fMRI measurements is about 0.7. The highest possible correlation you could expect from those two is the square root of their product, or 0.74. Problem is. . .a number of papers, including ones that get the big press, show correlations much higher than that. Which is impossible.
The Neurocritic blog has more details on this. What seems to have happened is that many researchers found signals in their patients that correlated with the behavior that they were studying, and then used that same set of data to compute the correlations between the subjects. I find, by watching people go by the in the street, that I can pick out a set of people who wear bright red jackets and have ugly haircuts. Herding them together and rating them on the redness of their attire and the heinousness of their hair, I find a notably strong correlation! Clearly, there is an underlying fashion deficiency that leads to both behaviors. Or people had their hair in their eyes when they bought their clothes. Further studies are indicated.
No, you can't do it like that. A selection error of that sort could let you relate anything to anything. The authors of the paper (Edward Vul and Nancy Kanwisher of MIT) have done the field a great favor by pointing this out. You can read how the field is taking the advice here.
+ TrackBacks (0) | Category: Biological News | Clinical Trials | The Central Nervous System
POST A COMMENT
- RELATED ENTRIES
- How Not to Do It: NMR Magnets
- Allergan Escapes Valeant
- Vytorin Actually Works
- Fatalities at DuPont
- The New York TImes on Drug Discovery
- How Are Things at Princeton?
- Phage-Derived Catalysts
- Our Most Snorted-At Papers This Month. . .