Here's the cry of someone who's been jerked around by too many journal referee reports. Hidde Ploegh of the Whitehead Institute has a piece in Nature News called "End the Wasteful Tyranny of Reviewer Experiments". That could have just possibly have been phrased more diplomatically, but I know what he's talking about.
Too often, reviewers try to show that they're fulfilling their responsibilities by requesting additional work from the authors of a paper under consideration. This happens more and more as you move up the hierarchy of journals, as both the novelty of the work and the incentive to publish it increase. No one's going to exert themselves too much to get their paper into Acta Retracta, even if some rogue reviewer were to try it, but Science and Nature (among others) can really make you perform tricks.
What this reminds me of is a story about Steve Wozniak, of Apple fame. When he was in college, his dorm had an old TV down in the lobby with a rabbit-ear antenna, which had to be messed with constantly to get a good picture. Woz apparently built a gizmo to fuzz out the reception, and used to sit inconspicuously in the back of the room, trying to see what sort of crazy positions he could twist people into as they held the antenna in what was seemingly the One Perfect Spot.
The referee equivalent is Just One More Experiment, and it's not always justified:
Submit a biomedical-research paper to Nature or other high-profile journals, and a common recommendation often comes back from referees: perform additional experiments. Although such extra work can provide important support for the results being presented, all too frequently it represents instead an entirely new phase of the project, or does not extend the reach of what is reported. It is often expensive and unnecessary, and slows the pace of research to a crawl. Among scientists in my field, there is growing concern that escalating demands by reviewers for the top journals, combined with the increasingly managerial role assigned to editors, now represents a serious flaw in the process of peer review.
Ploegh's point is that too many referees aren't reviewing the paper that they have; they're suggesting a whole new project or phase of research. And some of these wouldn't even affect the results and conclusions of the paper under review very much - they're just "Gosh, wouldn't it be nice if you would also. . ." experiments. The benefit for science, he says, is nowhere near commensurate with the disadvantage of holding up publication, messing with the career prospects of younger investigators, spending extra time and grant money, and so on. His suggestion?
The scientific community should rethink how manuscripts are reviewed. Referees should be instructed to assess the work in front of them, not what they think should be the next phase of the project. They should provide unimpeachable arguments that, where appropriate, demonstrate the study's lack of novelty or probable impact, or that lay bare flawed logic or unwarranted conclusions.
He also suggests that reviewers provide an estimate of the time and cost involved for their suggested experiments, and compare that to their purported benefits. I wouldn't mind seeing editors crack down on this some, either. I've had useful feedback with my own manuscripts, which had identified things that really did need to be shored up. But submitting a paper should not routinely be an exercise in having other people tell you what experiments you should run before you can publish your. When there really is a gap or flaw, naturally, it's appropriate to ask for more, but I agree with Ploegh that a reviewer needs to make a case for such things, rather than just asking for them as a matter of routine.
Ploegh has a larger historical point to make as well. Looking back at the earlier days of , say, molecular biology, you get the impression that if someone sent in an interesting paper that seemed reasonable, it would just get published, without all these trips back to the bench. Somehow, the mechanics of science (and especially scientific publication) have changed. Has it been for the better? Or would we all be better off letting more things through as they stand, if they're clearly presented and logically consistent?
I wonder if journals might consider publishing in this style, while then adding an editorial note about what further experiments had been suggested by reviewers. This would fulfill the function of pointing out potential weak points or areas for further exploration, but without delaying things so much. I don't see this happening - but why not, exactly?