I have a larger comment, sparked by the controversy over the NASA-arsenic-bacteria paper in Science. But it's not just about that one. It's about the "reactome" paper (also in Science and now retracted), the hexacyclinol synthesis published in Ang. Chem., and others. There have been, I think it's fair to say, a number of very arguable papers published in very high-profile journals in recent years. What's going on?
I want to make it clear that I'm not upset about journals published "out-there" work. In fact, I wish that there were a bit more of it. But at the same time, if you're going to go out there on the edge, you'd better have some solid stuff to report when you come back and write up the paper. Extraordinary claims really do require extraordinary evidence, and that's where things seem to be breaking down.
Peer review is supposed to catch these things. That reactome paper had chemists rolling their eyes as soon as they saw the synthetic schemes in it, and asking if anyone at the journal had thought to call someone who knew organic chemistry during the review process. This latest arsenic paper has other specialists upset, for different reasons (and, to be sure, for reasons that don't require much scientific specialization at all, as detailed in my post after I'd given the paper a close reading). But that hexacyclinol paper appeared in a chemistry journal, and had (one assumes!) been reviewed by competent chemists. How, then, could it have been published to immediate howls of derision about the quality of the evidence in it?
I also want to make clear that I'm not talking about some of the other categories of bad papers, such as the things are are probably true, but of little interest to anyone. And in the probably-not-true category, lower-ranking journals let not-so-good stuff through pretty often. I've been hard on Biorganic and Medicinal Chemistry Letters here before, among other journals, for publishing things that appear to have been incompetently reviewed. But these journals aren't Science or Nature, and the whole point of prestigious journals is that the things that appear in them are supposed to be important, and they're also supposed to be thoroughly vetted.
Is it the push to land the big papers that will make a big splash? Does that cause people in the editorial offices to bend the rules a bit? The official answer from every journal editor that's ever lived to such questions has been "Of course not!", but you have to wonder. Is it a problem with how they're assigning papers for review - who they go to, or how seriously the reviews are taken when they come back? I really don't know. I just know that we seem to be seeing a lot of embarrassing stuff in the literature these days. It's not supposed to work that way.