« The 2013 Drug Approvals: Not So Great? |
| Adoptive T-Cell Therapy for Cancer: The Short Version »
January 7, 2014
How Much Is Wrong?
Here's another take, from Jeff Leek at Simply Statistics, on the "How much published research is false?" topic. This one is (deliberately) trying to cut down on the alarm bells and flashing red lights.
Note that the author is a statistician, and the the arguments made are from that perspective. For example, the Amgen paper on problems with reproducibility of drug target papers is quickly dismissed with the phrase "This is not a scientific paper" (because it has no data), and the locus classicus of the false-research-results topic, the Ioannidis paper in PLoS Medicine, is seen off with the comment that "The paper contains no real data, it is purely based on conjecture and simulation."
I'll agree that we don't need to start assuming that everything is junk, as far as the eye can see. But I'm not as sanguine as Leek is, I think. Semi-anecdotal reports like the Amgen paper, the Bayer/Schering paper, and even scuttlebutt from Bruce Booth and the like are not statistically vetted scientific reports, true. But the way that they're all pointing in the same direction is suggestive. And it's worth keeping in mind that all of these parties have an interest in the answer being the opposite of what they're finding - we'd all like for the literature reports of great new targets and breakthroughs to be true.
The one report where Leek is glad about the mathematical underpinnings is the Many Labs project. But there's something about that that bothers me. The Many Labs people were trying to replicate results in experimental psychology, and while there's probably some relevance to the replications problems in biology and chemistry, there are big differences, too. I worry that everything is getting lumped together as Science, and if this part of Science is holding up, then those worries that people in other parts of Science have are probably ill-founded (after all, they don't have any real numbers, right?)
+ TrackBacks (0) | Category: The Scientific Literature
POST A COMMENT
- RELATED ENTRIES
- How Not to Do It: NMR Magnets
- Allergan Escapes Valeant
- Vytorin Actually Works
- Fatalities at DuPont
- The New York TImes on Drug Discovery
- How Are Things at Princeton?
- Phage-Derived Catalysts
- Our Most Snorted-At Papers This Month. . .