« Conference Travel, With Blogging |
Main
| From the RSC/SCI Symposium: A Med-Chem Anomaly »
September 12, 2011
The Scientific Literature Gets Kicked Around
Posted by Derek
It seems that the credibility of the scientific literature has been taking a beating recently. This has come about for several reasons, and through several different motivations. I'll get one of the most important out of the way first - politics. While this has been a problem for a long time, there's been a really regrettable tendency in US politics the last few years, a split across broadly left/right lines. Cultural and policy disagreements have led to many on the left claiming the Dispassionate Endorsement of Settled Science, while others on the right end up complaining that it's nothing of the sort, just political biases given a quick coat of paint. Readers will be able to sort several ongoing controversies into that framework.
Political wrangling keeps adding fuel to the can-we-trust-the-literature argument, but it would still be a big issue without it. Consider the headlines that the work of John Ioannidis draws. And there's the attention being paid to the number of retractions, suspicions of commercial bias in the medical literature, the problems of reproducibility of cutting-edge results, and to round it all off, several well-publicized cases of fraud. No, even after you subtract the political ax-grinding, there's a lot of concern left over (as there should be). There are some big medical and public policy decisions to be made based on what the scientific community has been able to figure out, so the first question to ask is whether we've really figured these things out or not.
A couple of recent articles prompted me to think about all this today. The Economist has a good overview of the Duke cancer biomarker scandal, with attention to the broader issues that it raises. And Ben Goldacre has this piece in The Guardian, highlighting this paper in Nature Neuroscience. It points out that far too many papers in the field are using improper statistics when comparing differences-between-differences. As everyone should realize, you can have a statistically significant effect under Condition A, and at the same time a lack of a statistically significant effect under Condition B on the same system. But that doesn't necessarily mean that the difference between using Condition A versus Condition B is statistically significant. You need to go further (usually ANOVA) to be able to say that. The submission guidelines for Nature Neuroscience itself make this clear, as do the guidelines for plenty of other journals. But it appears that a huge number of authors go right ahead and draw the statistically invalid comparison anyway, which means that the referees and editors aren't catching it, either. This is not the sort of thing that builds confidence.
So the questions about the reliability of the literature are going to continue, with things like this to keep everyone slapping their foreheads. One can hope that we'll end up with better, more reliable publications when all this is over. But will it ever really be over?
Comments (10)
+ TrackBacks (0) | Category: The Scientific Literature
POST A COMMENT
- RELATED ENTRIES
- › Chemistry Software Questions Here
- › An Engineered Rhodium-Enzyme Catalyst
- › Chemistry Software Questions Answered Monday
- › More on That Crowdsourced CNS Research
- › Thanksgiving Synthesis
- › TauRx Goes Into Phase III For Alzheimer's
- › The Galaxy Is Full of Gunk
- › And Since We're Talking About Insider Trading
1. anon on September 12, 2011 5:39 AM writes...
WRT politics, for the democrat/NYT war on science you just need to consider vaccines. Or the health supplement of the day being fawned over on 60 minutes. All politicians (and their journalist lackeys) are self-serving liars, so there is no reason to hold solely one party up as the paragon of the-evilness-which-bothers-me-personally-today.
Permalink to Comment2. Gregor on September 12, 2011 7:41 AM writes...
Science without integrity is worthless. No amount of credentials is a substitute for integrity. That is the heart of the matter.
Permalink to Comment3. simpl on September 12, 2011 9:55 AM writes...
The statistics misinterpretation is about depth of understanding, not morals. Don't you remember Steve Jay Gould's lifetime of preaching about an anthropocentric interpretation of genetics? Haven't you had cases where correlations were confused with causes? If you think the questions through, an isolated speculation can be mentioned in discussion as something for next time. It can aso improve your team's experimental design skills.
Permalink to Comment4. luysii on September 12, 2011 11:03 AM writes...
#3 Simpl -- It hasn't been resolved as yet, but Steven Jay Gould may turn out to have feet of clay along these exact lines. For details see http://luysii.wordpress.com/2011/06/26/hoisting-steven-j-gould-by-his-own-petard/
Permalink to Comment5. johnnyboy on September 12, 2011 12:44 PM writes...
@3: I don't think a question of dishonesty was raised for this particular issue (if Goldacre does, he's going a bit too far on his crusade). But Derek is right that this should be caught by reviewers, especially if it's the journal's guidelines for authors - unfortunately I doubt that reviewers in general spend a lot of time checking statistical methodology. From what i've seen, the overall feeling of biology/medical researchers when it comes to statistical testing lies somewhere within the spectrum between annoyance and horror.
Permalink to Comment6. anchor on September 12, 2011 1:53 PM writes...
#5 ...I believe that reviewers.."misinterpretation is about depth of understanding." I suppose #3 put it nicely!
Permalink to Comment7. Matt D on September 12, 2011 2:00 PM writes...
#1: False equivalence. Climate change denial and belief in creationism are becoming litmus tests for serious candidacy in the Republican party. I don't see many Democratic politicians proudly proclaiming an anti-vaccination stance as a prerequisite for election.
Permalink to Comment8. ChemBob on September 12, 2011 2:02 PM writes...
I recently submitted a paper that was rejected for the following reasons(in addition to some legitimate criticisms in which case further experiments or additional research was conducted to clarify and mitigate such concerns)
1. experiments that a reviewer said were inconclusive but that we had never actually run. In this case I agree, the results of such experiments would have been worthless and so were not even tried and were not described.
2. Using compounds that had no connection with the effects being studied i.e. the reveiwer claims that we used compounds that I have never touched except to do chemical inventory and were not mentioned in the actual manuscript.
3. Insufficient reading of the experimental section in my opinion as comments were made indicating a lack of understanding on how the system was actually synthesized.
Look, I know personal bias means that we understand our work better than anyone else and that when people in our field read the work some things appear obvious to us that are not apparent to people outside the field. I admit, the paper wasn't and isn't the newest thing or the highest impacting study to come out in this field, but how can we judge if we don't read these things thoroughly? Is this where scientific publishing is going, I'd like to think not, I still have a long career ahead of me.
Permalink to Comment9. MIMD on September 12, 2011 9:56 PM writes...
Meta-problem with "contaminated" literature e.g., bu ghostwriting. How do we know what's "real?"
See here.
Permalink to Comment10. fred on September 14, 2011 5:57 PM writes...
There is nothing recent about these problems. What is recent is that people are beginning to talk about it. I once had a paper rejected because a reviewer said, among other things, that I had used a flawed SAS module for the statistical analysis. The reviewer said that the signs always came out wrong in SAS.
The fact that I could explain why the reviewer was incorrect about the SAS results didn't seem to matter.
Permalink to Comment