Corante

About this Author
DBL%20Hendrix%20small.png College chemistry, 1983

Derek Lowe The 2002 Model

Dbl%20new%20portrait%20B%26W.png After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases. To contact Derek email him directly: derekb.lowe@gmail.com Twitter: Dereklowe

Chemistry and Drug Data: Drugbank
Emolecules
ChemSpider
Chempedia Lab
Synthetic Pages
Organic Chemistry Portal
PubChem
Not Voodoo
DailyMed
Druglib
Clinicaltrials.gov

Chemistry and Pharma Blogs:
Org Prep Daily
The Haystack
Kilomentor
A New Merck, Reviewed
Liberal Arts Chemistry
Electron Pusher
All Things Metathesis
C&E News Blogs
Chemiotics II
Chemical Space
Noel O'Blog
In Vivo Blog
Terra Sigilatta
BBSRC/Douglas Kell
ChemBark
Realizations in Biostatistics
Chemjobber
Pharmalot
ChemSpider Blog
Pharmagossip
Med-Chemist
Organic Chem - Education & Industry
Pharma Strategy Blog
No Name No Slogan
Practical Fragments
SimBioSys
The Curious Wavefunction
Natural Product Man
Fragment Literature
Chemistry World Blog
Synthetic Nature
Chemistry Blog
Synthesizing Ideas
Business|Bytes|Genes|Molecules
Eye on FDA
Chemical Forums
Depth-First
Symyx Blog
Sceptical Chymist
Lamentations on Chemistry
Computational Organic Chemistry
Mining Drugs
Henry Rzepa


Science Blogs and News:
Bad Science
The Loom
Uncertain Principles
Fierce Biotech
Blogs for Industry
Omics! Omics!
Young Female Scientist
Notional Slurry
Nobel Intent
SciTech Daily
Science Blog
FuturePundit
Aetiology
Gene Expression (I)
Gene Expression (II)
Sciencebase
Pharyngula
Adventures in Ethics and Science
Transterrestrial Musings
Slashdot Science
Cosmic Variance
Biology News Net


Medical Blogs
DB's Medical Rants
Science-Based Medicine
GruntDoc
Respectful Insolence
Diabetes Mine


Economics and Business
Marginal Revolution
The Volokh Conspiracy
Knowledge Problem


Politics / Current Events
Virginia Postrel
Instapundit
Belmont Club
Mickey Kaus


Belles Lettres
Uncouth Reflections
Arts and Letters Daily
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline

In the Pipeline

« RSS Readers: An Update | Main | The 2013 Medicine/Physiology Nobel: Traffic »

October 4, 2013

An Open Access Trash Heap

Email This Entry

Posted by Derek

Science magazine and writer John Bohannon have done us all a favor. There's a long article out in the latest issue that details how he wrote up a terrible, ridiculous scientific manuscript, attached a made-up name to it under the aegis of a nonexistent institution, and sent this farrago off to over three hundred open-access journals. The result?

On 4 July, good news arrived in the inbox of Ocorrafoo Cobange, a biologist at the Wassee Institute of Medicine in Asmara. It was the official letter of acceptance for a paper he had submitted 2 months earlier to the Journal of Natural Pharmaceuticals, describing the anticancer properties of a chemical that Cobange had extracted from a lichen.

In fact, it should have been promptly rejected. Any reviewer with more than a high-school knowledge of chemistry and the ability to understand a basic data plot should have spotted the paper's short-comings immediately. Its experiments are so hopelessly flawed that the results are meaningless.

I know because I wrote the paper. Ocorrafoo Cobange does not exist, nor does the Wassee Institute of Medicine. Over the past 10 months, I have submitted 304 versions of the wonder drug paper to open-access journals. More than half of the journals accepted the paper, failing to notice its fatal flaws. Beyond that headline result, the data from this sting operation reveal the contours of an emerging Wild West in academic publishing.

Well, sure, you're saying. Given the sorts of lowlife publishers out there, of course they took it, as long as the check cleared. But it's even worse than it appears:

Acceptance was the norm, not the exception. The paper was accepted by journals hosted by industry titans Sage and Elsevier. The paper was accepted by journals published by prestigious academic institutions such as Kobe University in Japan. It was accepted by scholarly society journals. It was even accepted by journals for which the paper's topic was utterly inappropriate, such as the Journal of Experimental & Clinical Assisted Reproduction.

Here's all the documentation, and it documents a sorry state indeed. You'll note from the world map in that link that India glows like a fireplace in this business. Nigeria has a prominence that it does not attain in the legitimate science publishing world, and there are exotic destinations like Oman and the Seychelles to be had as well. The editors of these "journals" tend to be people you've never heard of from universities that you didn't even know existed. And the editorial boards and lists of reviewers have plenty of those folks, mixed in with people who reviewed one paper before they didn't know better, and with people who didn't realize that their names were on the mastheads at all.

Bohannon didn't actually submit the exact same manuscript to all 304. He generated mix-and-match versions using an underlying template, giving him variations of the same crap and taking great care that the resulting papers should be obviously flawed:

he papers describe a simple test of whether cancer cells grow more slowly in a test tube when treated with increasing concentrations of a molecule. In a second experiment, the cells were also treated with increasing doses of radiation to simulate cancer radiotherapy. The data are the same across papers, and so are the conclusions: The molecule is a powerful inhibitor of cancer cell growth, and it increases the sensitivity of cancer cells to radiotherapy.

There are numerous red flags in the papers, with the most obvious in the first data plot. The graph's caption claims that it shows a "dose-dependent" effect on cell growth—the paper's linchpin result—but the data clearly show the opposite. The molecule is tested across a staggering five orders of magnitude of concentrations, all the way down to picomolar levels. And yet, the effect on the cells is modest and identical at every concentration.

One glance at the paper's Materials & Methods section reveals the obvious explanation for this outlandish result. The molecule was dissolved in a buffer containing an unusually large amount of ethanol. The control group of cells should have been treated with the same buffer, but they were not. Thus, the molecule's observed "effect" on cell growth is nothing more than the well-known cytotoxic effect of alcohol.

The second experiment is more outrageous. The control cells were not exposed to any radiation at all. So the observed "interactive effect" is nothing more than the standard inhibition of cell growth by radiation. Indeed, it would be impossible to conclude anything from this experiment.

I like this - the paper looks superficially presentable, but if you actually read it, then it's worthless. And yes, I realize that I've described a reasonable fraction of the ones that actually get published, but this is a more egregious example. I hope. The protocol was that Bohannon submitted the paper, and if it was rejected outright, that was that. If any reply came back addressing the paper's flaws in any way, he had a version ready to send back with more stuff in it, but without fixing any of the underlying problems. And if the paper was accepted, at any point in the process, he sent the journal a note that they'd discovered a serious flaw in their work and had to withdraw the manuscript.

157 journals accepted the paper, and 98 rejected it. He'd submitted it to a further 49 journals from his original list, but at least 29 of those appeared to be out of the business entirely, and the other 20 still had the paper "under review". So of those 255 decisions, 149 of them looked as if they'd occurred with little or no review. For a rejection, that's not so bad - this is a perfect example of manuscript that should not even be sent out for review. But the acceptances, well. . .

The other 106 editorial decisions made with some review are problematic, too. 70% of these were acceptances. Even in the few cases (36 times out of 304) where the paper was actually reviewed, and the reviewers realized that something was wrong with it (as they should have), the paper was accepted by 16 journals anyway.

The Elsevier journal that took this heap of junk was Drug Invention Today, in case you're wondering. I've never heard of it, and now I know why. The Sage journal was the Journal of International Medical Research, so you can strike that one off your list, too, assuming that the name wasn't enough all by itself. Another big open-access publisher, Hindawi (they advertise on the back cover of Chemistry World in the UK) rejected the paper from two of its journals, much to their relief. Jeffrey Beall's list of predatory publishers came as as pretty accurate, as well it might.

The problems with all this are obvious. These people are ripping off their authors for whatever publication fees they can scam, and some of these authors are not in a position to afford the treatment they're getting. No doubt some subset of the people who send manuscripts to these places are cynically padding their publication lists. The "editors" of these things get to reap a little unearned prestige for their "efforts" as well, so the whole enterprise just adds to the number of self-inflated jackasses with padded reputations, and the world is infested with too many of those people already. But I'm sure that there's another subset of authors who don't realize that they're submitting their results into a compost pile, and being asked to pay for the privilege. The first group are contemptible; the second group is sad. None of it's good.

Comments (57) + TrackBacks (0) | Category: The Dark Side | The Scientific Literature


COMMENTS

1. Hap on October 4, 2013 8:35 AM writes...

It was even accepted by journals for which the paper's topic was utterly inappropriate, such as the Journal of Experimental & Clinical Assisted Reproduction.

That sounds like a journal not safe for work in two ways instead of just the one suggested by the title. If it were in your professional field, I'm really sorry - reading a journal shouldn't require a home computer, a machete, and a shovel.

Permalink to Comment

2. Janne on October 4, 2013 8:38 AM writes...

Journalist sends of junk paper to low-ranking journals. Most low-ranking journals accept it. Which is why they're low-ranking to begin with of course. Big surprise.

For this to be about open access in any way he would have had to also send it out to similarly ranked closed journals and look for a significant difference between them. I am willing to bet he would not find any such difference.

To quote myself in another forum:

"Sloppy work, in other words, with a badly designed experiment with no controls, and sensationalist conclusions that don't follow from the results. The kind of work that a reputable research journal would quickly reject. Except, apparently, when it's in the "News" section."

Permalink to Comment

3. Anonymous on October 4, 2013 8:40 AM writes...

Well this puts to bed the notion that this is only a problem in China. This is absolutely pandemic, and horrifying.

I guess/hope that with this issues being exposed in this manner, there will be much more scrutiny of people's publication records by recruiters and hiring managers, so that those with dodgy papers will quickly be eliminated from consideration, and this "black science market" will be killed off.

Permalink to Comment

4. road on October 4, 2013 8:44 AM writes...

News flash: crappy journals publish crappy papers!

Nothing to see here...

Permalink to Comment

5. Anonymous on October 4, 2013 8:50 AM writes...

"News flash: crappy journals publish crappy papers!"

I agree, but crappy journals publish complete lies that could be spotted by a high school scientist? And so many of them?

Now that is news!

Permalink to Comment

6. MoMo on October 4, 2013 8:53 AM writes...

He should of kept this quiet- he probably would have gotten collaborative research offers from Pharma.

Permalink to Comment

7. Teddy Z on October 4, 2013 8:56 AM writes...

Funny, I posted on a related topic this morning...Conferences in China using the likenesses of those not even speaking to attract attendeess.

Permalink to Comment

8. Virgil on October 4, 2013 9:00 AM writes...

As is being Twittered and blogged elsewhere, there are some pretty HUGE problems with this, which are being missed by the mainstream scientific media in its reporting....

1) It's a news piece by a journalist, NOT a peer reviewed scientific paper.

2) There was no "control" group (i.e., non open access, non predatory journals). I would bet this would get into some higher up journals if only he'd tried.

3) There's no analysis whatsover - did the acceptance correlate with impact factor for example? What was the average review time? How many reviewers (just one, just editorial review, or more).

4) This is not about open access, that just happens to be in the title.

5) This borders on human subjects/outcomes research, but the author is not at a university, does not have IRB approval, did not get any kind of informed consent from the persons involved. As such, it's easy to see why this would be inadmissible as a research paper.

If you or I had done this and tried to get it published in Science, we would be crucified, but apparently if you're a journalist then the standards don't apply. So yeah, maybe if the headline was "hoity toity old media giant, feeling the pressure from open access, tried to pull one over on their new business rivals", then I'd be more inclined to believe it.

Permalink to Comment

9. Curious Wavefunction on October 4, 2013 9:09 AM writes...

I have lost count of how many "journals" have emailed me for an opportunity to publish in their august pages; being clearly uninterested in career advancement I turned these attractive offers down.

This is a scandal indeed, and it affects lesser known, struggling researchers in developing countries more than anyone else. Ironically as the experiment shows, most of the scandal is run out of the developing countries themselves so they are really shooting themselves in the foot here.

#8: Just because a story is not published in peer-reviewed journals does not mean it's not valid. By that token every story in the New Yorker would be considered to have giant holes in it. Yes, the standards for journalists and scientists are different, but that does not make journalists' work any less valid or important.

Permalink to Comment

10. Anonymous on October 4, 2013 9:23 AM writes...

I agree with Virgil. Note that this study has been published in Science WITHOUT a control experiment. The latter should have been done by submitting the bogus manuscript to the same number of non-OA journals. What is this telling us about the scientific competence of editors of high-impact journals?

I feel like submitting the same article signed by the most famous scientist in that field. Any bet on what percentage of respectable journals would accept it with minor corrections?

Permalink to Comment

11. a. nonymaus on October 4, 2013 9:48 AM writes...

"The paper was accepted by journals hosted by industry titans Sage and Elsevier."

Elsevier up to dodgy antics? I'm shocked and must lay down for a bit; take me to the fainting couch.

Permalink to Comment

12. Anonymous on October 4, 2013 10:06 AM writes...

If you have ever had an online "discussion" of climate change, or evolution or vaccines or...ad infinitum, then you have run into these journals, that get trotted out as "peer reviewed". Yes I agree that there are problems with the authors methodologies, but the point is well known among those of us that comment on some of the aforementioned topics.

Permalink to Comment

13. The Iron Chemist on October 4, 2013 10:44 AM writes...

"But I'm sure that there's another subset of authors who don't realize that they're submitting their results into a compost pile, and being asked to pay for the privilege."

It's a bit like hearing about people falling for a Nigerian email scam. At some point, you have to wonder how on earth these people could possibly be suckered into this.

Permalink to Comment

14. Anonymous on October 4, 2013 10:47 AM writes...

The thing that gets me most is that you could use this 'peer-reviewed publication' to market your bogus cancer treatment to desperate patients.

Permalink to Comment

15. Van Gogh on October 4, 2013 11:07 AM writes...

I also wonder which kind of comments it got from the reviewers: did they ask for straight rejection or polite "major revisions"? On which grounds, i.e. was the scam spotted some times?
Now, what I'm eagerly waiting for are 1- fraudulent papers copying-pasting it and 2- citations (it is published after all) to support findings from serious projects.

Permalink to Comment

16. TheBrummell on October 4, 2013 12:29 PM writes...

Virgil: "5) This borders on human subjects/outcomes research, but the author is not at a university, does not have IRB approval, did not get any kind of informed consent from the persons involved. As such, it's easy to see why this would be inadmissible as a research paper."

It does? I am woefully ignorant about IRB processes and what kind of study needs / does not need IRB approval because of human subjects. But I didn't see any human subjects or outcomes in the piece - unless you count the mentioned editors who spouted off about trust - completely missing the irony of their own words in the process.

Why would someone need to get IRB approval to expose fraud or other scams?

Permalink to Comment

17. John on October 4, 2013 12:56 PM writes...

I'm read this to my wife and her first response was: Wasn't that arsenic-based bacteria paper published in Science.

I think it's also important to mention that Science has a bit of an interest in establishing a difference between themselves and the plebeian-joournals.

Permalink to Comment

18. Scholander on October 4, 2013 1:08 PM writes...

@Van Gogh: They weren't actually published in the end. The fake author retracted it immediately after acceptance.

Permalink to Comment

19. Curt F. on October 4, 2013 1:46 PM writes...

This another fine example of fraud as a research technique. Or, to be more precise, a journalistic technique. As interesting as the data is, I find the manner in which it was obtained to be unethical and annoying because it involves intentional deception of the editors and manuscript reviewers of the research community. And the manner of the deception wastes the time these community members.

Many commenters here have probably spent time reviewing papers for journals. How would you like it if discovered a paper you were assigned to review was a fraud cooked up by a journalist to "sting" you?

The fact is, the editors and reviewers that spent time on these bogus manuscripts *are* human subjects, but they were *not* informed of the study; much less did they consent. Commenter Virgil is right that in many instances IRB approval would be necessary to conduct these sorts of studies. An example is Rand Ghayad's study last year on employer consideration of resumes of the long term unemployed. His study involved fraudulently submitting bogus resumes to various job postings. (Click my name-link to go to a blog post with discussion of this study.) Since he was at a University, he did have to obtain IRB approval for his study. It isn't clear to me how IRB approval was obtained, since in that case too, informed consent was not obtained from research subjects. I wrote to Northeastern University's IRB to ask them about these issues, but they never responded.

Permalink to Comment

20. Validated Target on October 4, 2013 1:52 PM writes...

@4 and 5: News flash: crappy journals publish crappy papers!

So-called "reputable" journals publish many totally crappy papers that wouldn't stand up to unbiased CRITICAL reviews but Editors can favor their pals and even overrule negative reviews. I am NOT talking about the famous cases (arsenic eating bacteria, etc.). I'm talking about worthless, poorly done studies with catchy titles and simplistic conclusions that will get cited but not read. I've seen it; it still happens; it makes me sick.

Permalink to Comment

21. Curt F. on October 4, 2013 1:56 PM writes...

@16. TheBrummell. Note that plenty of open access journals rejected his fraudulent articles. Does this guy Bohannon feel the least bit bad about wasting the time of those journals' editors, reviewers, and publishers?

If someone at a cocktail party tells you, "I cheat on my taxes ALL the time, EVERY year, and I NEVER get audited," does that really tell you more about the IRS or about the kind of person you're talking to?

Permalink to Comment

22. The Iron Chemist on October 4, 2013 2:00 PM writes...

Michael Eisen had an interesting response to all of this that also brought up the issue of what's going on in non-OA journals:

http://www.michaeleisen.org/blog/?p=1439

I'd agree with him in that there do seem to be lackadaisical, poorly informed, and/or over-burdened referees. What percentage of the referee pool this amounts to is open for debate. Since this hasn't been mentioned, I'd also add that there is likely a strong bias towards superstar researchers. I've seen countless molecules in high-profile journals that lack the fundamental characterization supposedly required for publication, even when the stuff is made in gram quantities.

I would, however, counter the whole "non-OA journals are as bad as OA journals" theme with the fact that the non-OA journals don't have as strong an incentive to accept EVERYTHING that gets sent in. Does a crappy paper get accepted due to the incompetence of reviewers or the dishonesty of the publisher?

Permalink to Comment

23. Objective on October 4, 2013 3:17 PM writes...

@8 and @10 - totally agree.

Regardless of where a paper is published, the AUTHOR and his/her manuscript are linked -- through their merit and integrity.


Journals like Science feel that
their existence is threatened. Even with peer-review -- their are lots of Bogus papers in Nature and Science.

One can envision a scenario where OPEN ACCESS journals are offering a platform for Authors to take responsibility for their reputation and publish "without peer-review". If the science merits -- it will automatically stand scrutiny over time. Authors will suffer bad reputation - if they publish garbage and this will bear itself out.

Permalink to Comment

24. TheBrummell on October 4, 2013 3:38 PM writes...

Curt (#19, #21), I still don't see it. I suppose the lying inherent in Bohannon's approach is on the wrong side of a fuzzy ethical line, but not by much. Perhaps some sort of oversight would have been appropriate.

But, he also points out that he strongly suspects very little time was actually wasted, due to the obvious nature of the flaws he put into the spoof paper. The ethanol thing stands out to me - anybody, even somebody with zero training in chemistry or biology, should be able to at least suggest in a critical review that perhaps the lack of effect (note the lack of effect, apparent in the figure but not in the text of the spoof paper, is itself a giant red flag) is due to the widely-known toxic effects of ethanol.

This is not a case of hiding clever easter eggs deep in a paper trying to trip up a group of basically honest and competent scientists and editors, it's handing someone a bag of garbage with the word "candy" painted on the side.

And I don't see the parallel with some random person at a cocktail party who tells me about their taxes. That part reads like a non-sequitor to me.

Permalink to Comment

25. MoMo on October 4, 2013 3:55 PM writes...

I don't buy your vitriolic argument Curt F.,

The fact that a bright journalist caught them all off-guard with their ink-stained greedy money-grubbing mechanisms and hands outstreched is a breath of fresh air.

Journals need policed while were fleeced.

Permalink to Comment

26. partial agonist on October 4, 2013 4:24 PM writes...

Complain all you want about the reputable journals, but I have to think that most of them would at least catch a fake name, a fake department, in a fake school,

somewhere in the process.

Permalink to Comment

27. dave w on October 4, 2013 4:55 PM writes...

Seems to me that the concept of "publication" with respect to scientific papers is
obsolescent... the original value of a journal was to take manuscripts and print
them up for physical distribution; now that one can post a PDF on the internet (and
set up for "blog comments" if desired), the only value still added by the traditional
model is the prestige of "Publication in a Reviewed Journal": and the sort of observation
presented here suggests that this value is often illusory.

(In other words, if the journals - as presently implemented - were to disappear,
the main impact would be less to the actual ability to present and distribute
information to each other, than to the ability to claim brownie points for one's
"publication count"...)

-dw

Permalink to Comment

28. Curt F. on October 4, 2013 9:13 PM writes...

MoMo, thanks for the reply. As you can maybe guess, I think your argument is more than a bit vitriolic too. But even so, let's say your right, and that every journal that was caught publishing the fraudulent papers is just an example of "ink-stained greedy money-grubbing mechanisms and hands outstreched".

You realize that still leaves at least 98 journals that rejected the paper, or about a third of the total, right? 98 journals that wasted their time and effort rejecting Bohannon's self-professed fraud. For example, PLoS One rejected the paper. Do you think the PLoS reviewers would say that Bohannon has "done us all a favor"? I don't.

Permalink to Comment

29. Wile E. Coyote, Genius on October 4, 2013 9:16 PM writes...

I don't see the IRB angle as some have mentioned. Does Google get IRB approval in collecting data from human subjects? Does a political polling company have to get IRB approval to do a poll? Does a teacher have to have IRB approval to test his/her subjects? Lots of other examples that could demonstrate absurdity reductio absurdum (my Latin probably isn't tip top). It seems to me that IRB only applies to medical intervention. IRB approval for surveying the degree of fraud in the world is bunk. There is no medical intervention going on here. Ticked off editors for wasting time? Likely.

Permalink to Comment

30. Nuclear Option on October 5, 2013 1:08 AM writes...

John Bohannon = Evridiki Gerou? Just sayin.

Permalink to Comment

31. meagain on October 5, 2013 3:46 AM writes...

I come to this with no great surprise, but then I've no affection for the open publishing (author pays) model, and I don't buy the argument that somehow the internet makes reader-pay journals obsolete. It's true that Science has a vested interest, but we all have a vested interest in proper peer review.

Permalink to Comment

32. Anonymous on October 5, 2013 4:52 AM writes...

"Do you think the PLoS reviewers would say that Bohannon has "done us all a favor"?"

Yes, absolutely, the PLoS reviewers will be delighted to have such public endorsement of the quality of their work to show that it stands way above the rest of the crap that is out there. It validates their otherwise hidden efforts, and shows people that what they do is actually very important.

This is no different from a food critic going into a PLoS "restaurant" under cover, and writing a great review of the quality of their food. If I was a PLoS reviewer I would be delighted.

Permalink to Comment

33. Janne on October 5, 2013 5:47 AM writes...

#31: "[...] I've no affection for the open publishing (author pays) model,"

Author pays is not restricted to open access publications. Many closed access journals — including some highly ranked ones — also require page charges, and steep ones. A study a few years ago determined that at the time the average charges were actually higher for closed journals than for open access ones.

Permalink to Comment

34. Anonymous on October 5, 2013 5:55 AM writes...

Perhaps we should all just publish our papers here at In The Pipeline and review in the comments?

Permalink to Comment

35. Anonymous on October 5, 2013 12:14 PM writes...

I think I just found another spoof paper directly insulting the standards of modern science publishing:
http://www.ncbi.nlm.nih.gov/pubmed/24093096
Authors main thesis is "Upregulated expression of H3K27me3 protein". There is no such thing. There are many variants of histone 3 which can be expressed. H3K27me3 is post-translation modification. It can not be expressed. And this paper should not be accepted.
Alas, it is most likely not a spoof but an "important contribution to Scince" by Guangzhou university(ies)

Permalink to Comment

36. Curt F. on October 5, 2013 3:54 PM writes...

@32. Anonymous. I understand where you are coming from but you're a restaurant reviewer is in apt. Restaurant reviewers pay for their meal. The restaurant is compensated for serving the meal, and it presumably the wages of the employees required to deliver the service. The review is incidental to the transaction, which both the restaurant and secret reviewer agree to ahead of time.

Submitting papers to OA journals is not like that. You don't pay until it's time to publish. All the work of most journal editors and all journal reviewers is done on a volunteer basis anyway. They do the work out of a desire to serve a particular research community. A core assumption is that this community will submit manuscripts in good faith, which is obviously not the way that Bohannon and Science News like to roll.

Your analogy would for better if instead of a restaurant, it was dinner at a friend's house.

Permalink to Comment

37. Curt F. on October 5, 2013 3:58 PM writes...

Well I wrote that last comment on my phone. Sorry to everyone for the many many errors.

Permalink to Comment

38. SynChemSyn on October 5, 2013 3:58 PM writes...

No surprise whatsoever - just a waste of time for reviewers and others involved. What is flawed is the pseudo-scientific several-page article on the subject published in Science.

As for awfully poor, entirely fabricated papers, the best example I know is the fictional synthesis of hexacyclinol published by La Clair in early 2006 (Angew. Chem. Int. Ed. 2006, 45, 2769).

Even though the fraud was obvious and fully exposed everywhere within weeks after the article was published, I just realized that it took Angewandte six and a half years!! before eventually retracting it (Angew. Chem. Int. Ed. 2012, 51, 11647).

Permalink to Comment

39. aoeu on October 5, 2013 4:58 PM writes...

@Curt F.: While we're on the subject of wasting time, let's talk about the countless hours wasted by researchers placing their trust in published, supposedly peer-reviewed work that, for some reason, doesn't seem to work now that you're trying it.

What a huge waste of time it is when I find an experimental procedure in a paper that passed peer review, try it myself, and find that my results are nothing like the author's.

The (in most cases brief) inconvenience that these fake papers caused for some 300 reviewers pales in comparison to all the time wasted by probably every graduate student since publishing became a thing, who angrily scanned through a lazily peer-reviewed paper trying to figure out why the reaction isn't working like it says it should.

This editorial is not rigorous and should not be treated as such, but I will in the future very much appreciate how much time i'm saving by avoiding the journals that accepted this obviously flawed paper.

Permalink to Comment

40. Nat on October 6, 2013 5:20 AM writes...

That's a lot of work to stroke one's own ego. Some places look at the name and subject and accept the paper, some look at the check. Same end result, to make lost of money

Permalink to Comment

41. Anonymous on October 6, 2013 5:38 AM writes...

@36: True, that's a fair point, but...

@39: That's an even better point than #36.

Permalink to Comment

42. Anonymous on October 6, 2013 6:35 AM writes...

@41: I agree: This wasted no time for the journals that did not bother to review properly, while those that did spend time to review properly have got way than a few hours value worth of marketing, to show they are better quality than the majority.

I wonder how many of the journals went ahead and published anyway, even *after* the retraction. That would show the real worst ones...

Permalink to Comment

43. Scholander on October 6, 2013 8:46 AM writes...

dave w (#27): That makes an amazing amount of sense!

With inflation of value of publication count, it is literally impossible to keep track of the current literature. It would be amazing if "the literature" was following researchers' lab's progress online. Reliable journals like Science and Nature would adapt to become aggregators and news sites. (How many Science articles have you actually read all the way through this year? They already are broad trade journals for me.) Scientists could be quantifiably evaluated by traffic numbers and links to their work akin to a Google PageRanking rather than crude and flawed metrics like impact factor or number of pubs. The blog system seems ideal for scientific reporting, and it's a shame we missed the boat ten years ago.

I'm now willing to bet that within another ten years or so there will be another, more thorough sting like this that will finally pop the print science publication bubble. It's a shame, and a valid criticism, that this sting was only directed at open access, and not to science publishing in general.

Permalink to Comment

44. Ron on October 6, 2013 1:59 PM writes...

This is not surprising to me that in low-impact journals you'll find stuff that really has no merit/impact on science. Perhaps it's even completely fraudulent or the experiments were performed inappropriately.

However, there was a publication (Here is one study but this is not the one I had originally read as I'm not on my main computer: http://iai.asm.org/content/79/10/3855.full) that correlated an increase in retraction/fabrication with the higher impact factor of the journal.

I seriously wonder what would happen if the same type of article was submitted with a world-renowned scientist's name on it, with recommendations of individuals he felt should review it? What if two science worthy papers were submitted one by a no-name and one by a high name? It's well know there are often politics surrounding peer review of an article (especially in high impact journals), if the editor of Science denies this, he knows he'd be lying.

There are many problems with the current state of peer-reviewed publication system. I believe the first step towards correcting some of the issues is a double-blind review. A second suggestion is obtaining individuals who properly review a publication (this includes reading the materials and methods). How often do you think post-docs and graduate students are reviewing these? They know names of competitors/big labs. Do they even fully read and understand the methods sections (I know from experience the answer)? Let's not kid ourselves and say professors NEVER put peer-review burdens on post-docs and grad students, even though they're not skilled enough to review that paper. What about professors who perhaps extremely intelligent, are also not equipped to review certain papers due to their lack of methods?

In fairness we shouldn't stop there, we should make grants double-blind also. Even though the individual PI and the institute count for 40% of an NIH grant score, this could easily be performed by a different committee who never sees the actual grant.

As I said there are plenty of problems with the current system, but you're taking information from the editor of Science, which has it's own problems as well. Rather than poking a finger in a festering wound which we know exists, lets find a way to suture and bandage it and clear the current infection that is running through the body.

Permalink to Comment

45. Morten G on October 6, 2013 2:21 PM writes...

~300 submissions, ~150 acceptances, ~50 dead journals, ~10 with a submission fee, and ~100 rejections.

This may or may not be great advertisement for Open Access journals in general since I can't compare these numbers to Closed Access (are they called that?). But since the argument was raised that this was good advertising for the journals that rejected the papers I thought someone should list them.

Advances in Cancer: Research & Treatment
Neuroglia
Biochemical Journal
Cancer Growth and Metastasis
Chemotherapy Research and Practice
Carcinogenesis
Drugs and Therapy Studies
PhytoKeys
British Journal of Pharmaceutical Research
Journal of Postgraduate Medical Institute
The Journal of the Polish Biochemical Society
World Journal of Oncology
mBio
Plant Knowledge Journal
Annals of Cancer Research and Therapy
Journal of Health Informatics in Developing Countries
African Journal of Primary Health Care & Family Medicine
International Journal of Medical Biology
Current Chemistry Letters
Emerging Issues in Medical Diagnosis and Treatment
Cancers
Journal of Medical Sciences
Environmental Health Perspectives
Current Cellular Biochemistry
National Journal of Community Medicine
Frontiers in Pharmacology of Anti-Cancer Drugs
Stem Cells
Pakistan Journal of Medical Sciences
Studies in Mycology
Anti-Cancer Drugs
"Sustainability: Science, Practice, & Policy"
World Journal of Radiology
BioResources
International Journal of Research in Chemistry and Environment
Clinical and Molecular Hepatology
Oral & Maxillofacial Pathology Journal
Pharmaceutical Technology & Drug Research
Cancer and Clinical Oncology
Case Reports in Oncology
Biointerphases
Journal of Solid Tumors
JRSM Short Reports
International Journal of AgriScience
Mathematical and Computational Forestry & Natural-Resource Sciences
Neurosurgical Focus
Grand Rounds
Current Botany
Chemical Sciences Journal
Bulletin of the Korean Chemical Society
American Journal of Cancer Therapy and Pharmacology
Journal of Life Medicine
Iranian Journal of Immunology
ScienceJet
Insight Pharmaceutical Sciences
Open Medicine
American Journal of Nuclear Medicine and Molecular Imaging
Yonsei Medical Journal
Biodiversity Journal
International Journal of Biological Sciences
Algae
Immunome Research
Drugs in R&D
Western Journal of Emergency Medicine
Neoplasia
Journal of Tropical Agriculture
Interdisciplinary Bio Central
Phytotherapy Research
Journal of Sports Science and Medicine
Journal of Animal and Plant Sciences
Current Issues in Molecular Biology
International Journal of Pharmaceutical Sciences and Drug Research
Boreal Environment Research
"Engineering, Technology & Applied Science Research"
Acta Veterinaria Brno
Journal of Experimental and Applied Animal Sciences
BiO
International Journal of Agriculture and Biology
Israel Medical Association Journal
PLoS ONE
Acta Societatis Botanicorum Poloniae
Chinese Medical Journal
Radiology Case Reports
Caribbean Journal of Science
Folia Histochemica et Cytobiologica
Genes | Genomes | Genetics
Iranian Endodontic Journal
The Journal of Functional Foods in Health and  Disease
Laboratory Investigation
International Journal of Biomolecules and Biomedicine
Indo American Journal of Pharmaceutical Research
Archives of Medical Science
National Journal of Medical Research
Pakistan Journal of Botany
Natural Products against Cancer
Bosnian Journal of Basic Medical Sciences
International Journal of Engineering Science and Technology
Electronic Journal of Biotechnology
Physiological Research
Journal of Medical & Biological Sciences
Thammasat International Journal of Science and Technology
Current Research in Medicine and Medical Sciences
Biocell (Mendoza)
International Journal Of Advances in Engineering Science and Technology
International Journal of Advances in Biology
Journal of Liaquat University of Medical and Health Sciences
Pharmacognosy Journal
People's Journal of Scientific Research
Journal of Science And Enviromental Impact
Swiss Journal of Medical Science
International Journal of Agricultural and Biological Engineering
Experimental Agriculture & Horticulture
Electronic Journal of Polish Agricultural Universities
International Journal of Pharmaceutical and Biomedical Research
Journal of Ayub Medical College Abbottabad
Journal of Community Nutrition & Health
The Journal of Applied Research
Acta Biologica Cracoviensia
Scholars’ Research Journal

Journals that required a submission fee. Apparently Science weren't keen enough to pony up.

Acta Dermato-Venereologica
American Journal of Health Sciences
Singapore Medical Journal
International Journal of Advanced Pharmaceutical Science and Technology
theHealth
Journal of Environmental Biology
ePlasty
Medical Journal of Malaysia
Journal of Life Sciences
Journal of Physiology and Pharmacology

Permalink to Comment

46. bofh on October 7, 2013 1:29 AM writes...

@19: "Many commenters here have probably spent time reviewing papers for journals. How would you like it if discovered a paper you were assigned to review was a fraud cooked up by a journalist to "sting" you?"

Extremely relieved that the dreck I was reading was fake dreck by someone trying to see if the problem is as bad as everyone seems to delude themselves into thinking it isn't, as opposed to *actual* dreck.

Maybe I'm just extremely jaded from a lot of paper review, but if anyone here is unethical in my opinion it is those reviewers that don't do their job and allow such crap to slip by them into some peer-reviewed journal. I've wasted way, way more time reading worthless/pointless papers that should never have been published than I would reading one fake "sting" paper. And really, the only thing that said fake paper might ultimately do, is help make the flow of trash become less (more likely nothing will really happen at all, sadly).

While the whole OA vs. non-OA thing is somewhat of a pointless red herring and the conclusions about it are sketchy at best, the real conclusion of this study, which is worrying as shit, is the fact that there are a staggering amount of crappy journals out there. And that really benefits just about nobody. And sure, the fact that crappy journals exist isn't all that new, but what is new is the sheer absurd magnitude that it's been happening in as of late.

Permalink to Comment

47. Wibbler on October 7, 2013 4:37 AM writes...

As a researcher for a pharmaceutical company I have considered publishing in open access journals so that other scientists would not need a subscription to read the entire, hopefully useful, paper. Seeing these data I would now re-consider doing this. It seems that the whole area of open access journalism is crying out for some sort of accreditation?

Permalink to Comment

48. Buden of proof on October 7, 2013 7:05 AM writes...

Follow the money!

What motivation does anyone have for wanting to publish dreck in such open access fermenters? I'm sure there are many subtleties but I suspect the biggest is that they advance their careers in some form.

The onslaught will continue until promotion and review bodies, especially academic ones, reform how they handle publications. When medical schools, state colleges or higher prestige places publicly state that junk publications actually count against you then the tide will start to change. Synthetic papers should only count for career advancement after they have been replicated in some form such as a key reaction found to work in other's hands or yields that are not drastically out of line. Then, much as Wall Street is doing in some cases, put a 'claw back' provision in. If published work is later shown to be junk the institution should have rights to claw back from the offender's 401K an amount equal today's value of the original benefit. In this manner would authors have a stake in the long term future.

Journals that don't respond to junk and take 6 years to act on a retraction should also face sanction though it is less clear how to manage that.

Lastly, I strongly disagree with the point of view that suggests an IRB is needed. This is journalism and not science although it is journalism about science practices. It is probably not intended to come across this way, but Curt F.'s statements are from someone who views their time, and an editor's time, as just too important to waste. Really? Journals purport to provide a service to the community and earn income in the process. Reviewers *agree* to review for journals, they are not getting paid in cash but in ego-stroking instead. An editor or a reviewer that is incapable of seeing junk should not be doing it. In many of those journals I wonder just how close the editor is to the revenue stream? I suspect *very* close. You don't setup an IRB to catch lousy transmission repair shops. This is no different. They are preying on people, they deserve to be called out. Yes, the closed and traditional journals should also be tested but those are older and well established problems. At least the current investigation is catching up with a newer problem.

The burden of proof is with the journals -let's see what they all do.

Permalink to Comment

49. sbfren on October 7, 2013 9:13 AM writes...

An interesting take/discussion from Language Log, which includes some of the points that Virgil and bring up: http://languagelog.ldc.upenn.edu/nll/?p=7584

Permalink to Comment

50. MIMD on October 7, 2013 11:35 AM writes...

I don't think these matters address that other "little" problem...ghostwriting.

Permalink to Comment

51. Design Monkey on October 7, 2013 4:53 PM writes...

19. Curt F. on October 4, 2013 1:46 PM writes...
Many commenters here have probably spent time reviewing papers for journals. How would you like it if discovered a paper you were assigned to review was a fraud cooked up by a journalist to "sting" you?

---------------

I wouldn't especially mind. I've rejected papers that deserved it, and would reject obvious crap with use of flamethrower. No big deal, and it has even certain satisfaction in it.

Permalink to Comment

52. David Ross on October 9, 2013 4:29 AM writes...

I would like to point out that the Journal of International Medical Research (JIMR), published by SAGE, did not accept the article submitted to it for publication as it stands. It was accepted subject to further editorial queries being answered to the satisfaction of the journal. This is clearly laid out in the correspondence with the author.

JIMR is a long-established, respected journal that has been publishing quality research for 41 years. It is JCR-listed and its rejection rate was 62.5% in 2012. Professor Malcolm Lader Ph.D., M.D., F.R.C. Psych., F. Med. Sci., Emeritus Professor, King’s College London is the Editor of the journal and has been for the last 25 years. It has a world class Editorial Board.

We are extremely concerned that a paper with fundamental errors got through the initial review stage, and are taking steps to ensure that it cannot occur again, but are confident that the technical edit would have revealed the errors.

SAGE is committed to ensuring that the peer review and acceptance process for all our journals, whether traditional subscription-based or Open Access, is robust.

A fuller statement is posted on the SAGE website: http://tinyurl.com/jwldtgm.

David Ross
Executive Publisher, Open Access
SAGE

Permalink to Comment

53. dion on October 9, 2013 8:40 AM writes...

guess that SCIgen never existed, right?

Permalink to Comment

54. see on October 9, 2013 8:48 AM writes...

"SCIgen is a program created by scientists at the Massachusetts Institute of Technology that randomly generates nonsense in the form of computer science research papers, including graphs, diagrams, and citations. It uses a context-free grammar to form all elements of the papers, and its authors state that their aim is "to maximize amusement, rather than coherence."

"Students at Iran's Sharif University of Technology published a paper in the Journal of Applied Mathematics and Computation (which is published by Elsevier). The students wrote under the false, non-Persian surname, MosallahNejad, which translates literally from Persian, in spite of being "non-Persian" as: "from an Armed Breed". The paper was subsequently removed when the publishers were informed that it was a joke paper."
(...)
"A paper titled "Towards the Simulation of E-Commerce" by Herbert Schlangemann got accepted as a reviewed paper at the "International Conference on Computer Science and Software Engineering" (CSSE) and was briefly in the IEEE Xplore Database. The author is named after the Swedish short film Der Schlangemann. Furthermore the author was invited to be a session chair during the conference.
(...)
"A 2013 scientometrics paper demonstrated that at least 85 SCIgen papers have been published by IEEE."

From: http://en.wikipedia.org/wiki/SCIgen

Permalink to Comment

55. remember also on October 9, 2013 8:51 AM writes...

"Scientific publishing giant Elsevier put out a total of six publications between 2000 and 2005 that were sponsored by unnamed pharmaceutical companies and looked like peer reviewed medical journals, but did not disclose sponsorship, the company has admitted. Elsevier is conducting an "internal review" of its publishing practices after allegations came to light that the company produced a pharmaceutical company-funded publication in the early 2000s without disclosing that the "journal" was corp"

From: http://www.the-scientist.com/?articles.view/articleNo/27383/title/Elsevier-published-6-fake-journals/

Permalink to Comment

56. pete on November 19, 2013 12:08 PM writes...

@52 David Ross (Executive Publisher, Open Access
SAGE)

FYI: The SAGE link you list above is broken

Permalink to Comment

57. Doctors Loan on February 26, 2014 7:23 AM writes...

The article treats the issue fairly neutrally, but fails to bring up a key point that any discussion of American agriculture needs.
Finally, there are several interviewers, and you should have between 45% and 70% of hospitals opt out of obamacare your goal right there!
The problem is that the principle of doing several things simultaneously.

Permalink to Comment

POST A COMMENT




Remember Me?



EMAIL THIS ENTRY TO A FRIEND

Email this entry to:

Your email address:

Message (optional):




RELATED ENTRIES
How Not to Do It: NMR Magnets
Allergan Escapes Valeant
Vytorin Actually Works
Fatalities at DuPont
The New York TImes on Drug Discovery
How Are Things at Princeton?
Phage-Derived Catalysts
Our Most Snorted-At Papers This Month. . .