Thinking about the scientific literature, I got to wondering: which part of the chemistry literature is the most unreliable? (That is, fitting that "pessimist universe" model from my post the other day). There's been a recent bit of trouble in the organic chemistry world that I've been meaning to talk about, so I thought I'd lead up to it this way.
If you look at what organic chemistry papers have been retracted over the years, my strong impression is that most of them have been papers on new synthetic methods. That might make you think that this is the weak spot, but I'd disagree - I think that these stand out because people actually read them and try to reproduce them. An interesting new method is going to be tried out by others, and it'll be clear very quickly if there's a problem. I wrote about such a case on my old site, back in 2002:
". . .There's a paper out now (which for those of you in the field is Synthesis, 29, 2002), that belongs to the select group of articles whose sole purpose is to demolish another one. The now-discredited article was in the same journal over a year ago, presenting an interesting reaction that I thought we could make use of in my lab. (Keep in mind that the classic definition of "interesting" in the scientific literature is "interesting to me!") We actually tried the chemistry out. It flopped cleanly and completely, giving exactly the wrong product. I chalked it up to the weirdness of our current compounds, which is not to be underestimated. Some things work on them; some don't. We poured the reaction into the red waste can and did something else, which is one of the things I like most about medicinal chemistry.
The author of the latest article, though, had the same thing happen to him, and he didn't take it as quietly. Going back over the original examples, he shows that the published work won't, didn't, and can't go the way it was reported to. Some of this can be put in the "honest mistake" category, subheading "really sloppy honest mistakes," but I'm afraid, in the end, that some of it can't. . ."
Even things that don't look all that useful at the time will often get dragged out into the light by somebody eventually. No, my vote for unreliability is the papers on total synthesis of natural products. In almost all cases, no one will ever do those reactions again. The exceptions are the rare examples where the product is itself useful and the synthesis has industrial relevance (taxol!), or the times when more than one group is working on the molecule simultaneously and using similar approaches.
But how often do such things happen? Mostly, it's "Total Synthesis of Timewastotoxin", which either no one will ever see fit to make again, or if they do will have to use a completely different synthesis (because otherwise, what's the point?) The yield in step 48 could be inflated like a beach ball - who will ever know? Step 33 might be the only run that worked out of 64 tries. In fact, the pressures of such work make it rather likely that these very things have taken place.
I'm not saying that the majority of total syntheses in the literature are incorrect - since (in the end) the synthetic product can be compared against the natural one. Only outright fraud could explain papering over a problem that goes all the way to the last step. I don't think that that's very common at all, although I sure wouldn't bet on it never having happened. (When it has, it's most likely been due to a desperate grad student spiking a sample with the real product).
I supposed a better word than "unreliable" would be "indeterminate". Even though they're probably fine, no one knows if the advanced steps in a long synthesis are real or not. It's just not worth the trouble to find out. It's as if Schrodinger's cat stayed in a superposition of states because it was so hard to open the box, and because no one cared very much whether he was alive or dead anyway.