Corante

About this Author
DBL%20Hendrix%20small.png College chemistry, 1983

Derek Lowe The 2002 Model

Dbl%20new%20portrait%20B%26W.png After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases. To contact Derek email him directly: derekb.lowe@gmail.com Twitter: Dereklowe

Chemistry and Drug Data: Drugbank
Emolecules
ChemSpider
Chempedia Lab
Synthetic Pages
Organic Chemistry Portal
PubChem
Not Voodoo
DailyMed
Druglib
Clinicaltrials.gov

Chemistry and Pharma Blogs:
Org Prep Daily
The Haystack
Kilomentor
A New Merck, Reviewed
Liberal Arts Chemistry
Electron Pusher
All Things Metathesis
C&E News Blogs
Chemiotics II
Chemical Space
Noel O'Blog
In Vivo Blog
Terra Sigilatta
BBSRC/Douglas Kell
ChemBark
Realizations in Biostatistics
Chemjobber
Pharmalot
ChemSpider Blog
Pharmagossip
Med-Chemist
Organic Chem - Education & Industry
Pharma Strategy Blog
No Name No Slogan
Practical Fragments
SimBioSys
The Curious Wavefunction
Natural Product Man
Fragment Literature
Chemistry World Blog
Synthetic Nature
Chemistry Blog
Synthesizing Ideas
Business|Bytes|Genes|Molecules
Eye on FDA
Chemical Forums
Depth-First
Symyx Blog
Sceptical Chymist
Lamentations on Chemistry
Computational Organic Chemistry
Mining Drugs
Henry Rzepa


Science Blogs and News:
Bad Science
The Loom
Uncertain Principles
Fierce Biotech
Blogs for Industry
Omics! Omics!
Young Female Scientist
Notional Slurry
Nobel Intent
SciTech Daily
Science Blog
FuturePundit
Aetiology
Gene Expression (I)
Gene Expression (II)
Sciencebase
Pharyngula
Adventures in Ethics and Science
Transterrestrial Musings
Slashdot Science
Cosmic Variance
Biology News Net


Medical Blogs
DB's Medical Rants
Science-Based Medicine
GruntDoc
Respectful Insolence
Diabetes Mine


Economics and Business
Marginal Revolution
The Volokh Conspiracy
Knowledge Problem


Politics / Current Events
Virginia Postrel
Instapundit
Belmont Club
Mickey Kaus


Belles Lettres
Uncouth Reflections
Arts and Letters Daily
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline

In the Pipeline

« Lilly's Statin - Yes, It Is 2010 | Main | Fungal Structures to the Rescue »

June 24, 2010

All Those Worthless Papers

Email This Entry

Posted by Derek

That's what this article at the Chronicle of Higher Education could be called. Instead it's headlined "We Must Stop the Avalanche of Low-Quality Research". Which still gets the point across. Here you have it:

While brilliant and progressive research continues apace here and there, the amount of redundant, inconsequential, and outright poor research has swelled in recent decades, filling countless pages in journals and monographs. Consider this tally from Science two decades ago: Only 45 percent of the articles published in the 4,500 top scientific journals were cited within the first five years after publication. In recent years, the figure seems to have dropped further. In a 2009 article in Online Information Review, Péter Jacsó found that 40.6 percent of the articles published in the top science and social-science journals (the figures do not include the humanities) were cited in the period 2002 to 2006.

As a result, instead of contributing to knowledge in various disciplines, the increasing number of low-cited publications only adds to the bulk of words and numbers to be reviewed. Even if read, many articles that are not cited by anyone would seem to contain little useful information. . .

If anything, this underestimates things. Right next to the never-cited papers are the grievously undercited ones, most of whose referrals come courtesy of later papers published by the same damn lab. One rung further out of the pit are a few mutual admiration societies, where a few people cite each other, but no one else cares very much. And then, finally, you reach a level that has some apparent scientific oxygen in it.

The authors of this article are mostly concerned about the effect this has on academia, since all these papers have to be reviewed by somebody. Meanwhile, libraries find themselves straining to subscribe to all the journals, and working scientists find the literature harder and harder to effectively cover. So why do all these papers get written? One hardly has to ask:

The surest guarantee of integrity, peer review, falls under a debilitating crush of findings, for peer review can handle only so much material without breaking down. More isn't better. At some point, quality gives way to quantity.

Academic publication has passed that point in most, if not all, disciplines—in some fields by a long shot. For example, Physica A publishes some 3,000 pages each year. Why? Senior physics professors have well-financed labs with five to 10 Ph.D.-student researchers. Since the latter increasingly need more publications to compete for academic jobs, the number of published pages keeps climbing. . .

We can also lay off some blame onto the scientific publishers, who have responded to market conditions by starting new journals as quickly as they can manage to launch them. And while there have been good quality journals launched in the past few years, there have been a bunch of losers, too - and never forget, the advent of a good journal will soak up more of the worthwhile papers, lifting up the ever-expanding pool of mediocre stuff (and worse) by capillary action. You have to fill those pages somehow!

If this problem is driven largely by academia, that's where the solution will have to come from, too. The authors suggest several fixes: (1) limit job applications and tenure reviews to the top five or six papers that a person has to offer. (2) Prorate publication records by the quality of the journals that the papers appeared in. (3) Adopt length restrictions in printed journals, with the rest of the information to be had digitally.

I don't think that those are bad ideas at all - but the problem is, they're already more or less in effect. People should already know which journals are the better ones, and look askance at a publication record full of barking, arf-ing papers from the dog pound. Already, the best papers on a person's list count the most. And as for the size of printed journals, well. . .there are some journals that I read all the time whose printed versions I haven't seen in years.

No, these ideas are worthy, but they don't get to the real problem. It's not like all the crappy papers are coming from younger faculty who are bucking for tenure, you know. Plenty more are emitted by well-entrenched groups who just generate things that no one ever really wants to read. I think we've made it too possible for people to have whole scientific careers of complete mediocrity. I mean, what do you do, as a chemist, when you see another paper where someone found a reagent to dehydrate a primary amide to a nitrile? Did you read it? Of course not. Will you ever come back to it and use it? Not too likely, considering that there are eight hundred and sixty reagents that will already do that for you. We get complaints all the time about me-too drugs, but the me-too reaction problem is a real beast.

Now, I realize that by using the word "mediocrity" I'm in danger of confusing the issue. The abilities of scientists are distributed across a wide range - I doubt if it's a true normal distribution, but there are certainly people who are better and worse at this job. But I'm complaining on the absolute scale, rather than the relative scale. I know that there's always going to be a middle mass of scientific papers, from a middle mass of scientists: I just wish that the whole literature was of higher quality overall. A chunk of what now goes into the mid-tier journals should really be filling up the bottom-tier ones, and most of the stuff that goes into those shouldn't be getting done in the first place.

I suppose what bothers me is the number of people who aren't working up to their potential (although I don't always have the best position to argue that from myself!) Too many academic groups seem to me to work on problems that are beneath them. I know that limits in money and facilities keep some people from working on interesting things, but that's rare, compared to the number who'd just plain rather do something more predictable. And write predictable papers about it. Which no one reads.

Comments (40) + TrackBacks (0) | Category: The Scientific Literature | Who Discovers and Why


COMMENTS

1. Anon on June 24, 2010 8:34 AM writes...

I find this to be a fascinating topic and one that I end up discussing with my peers. There's some more comments on this topic by Peter Lawrence from Cambridge here:
http://www.int-res.com/articles/esep2008/8/e008p009.pdf

"Already, the best papers on a person's list count the most." But why do we need a giant list of publications if everyone already knows which publications are crappy ones? Perhaps if we limited job/tenure applications to 3 (or 4, or 5 whatever just pick a small number) and tell the applicant they can ONLY discuss what actually appears in those 3 papers then there would be pressure to actually write in-depth papers that contribute substantially to science rather than releasing results piecemeal in order to up one's publication count. Maybe not the magic bullet solution, but seems like it would help provide motivation for doing science the "correct" way.

What's interesting is that nobody questions what happens to those students that are working for a boss that actually does only write up quality publications for a finished/polished project. We all claim science should be done for science itself and not for one's the publication record, yet we still judge job applicants by their publication record. What if the applicant worked on a difficult project, made significant progress, but was unable to put the finishing touches on it before it's time to graduate and move on? If the boss doesn't publish it, since it's not fully finished yet, then the student that did honest hard work can't compete with their peers who worked for the publication machines.

I don't know what the solution is, but I think we all know what the problem is. Perhaps an easy place to start is to stop stigmatizing those without a lengthy publication list and let the work they did speak for itself. In other words, stop the attitude of no publications = no job interview.

Permalink to Comment

2. Duct Tape on June 24, 2010 8:36 AM writes...

The point regarding academic potential and 'fluff' within the publishing models are well taken. Nice post.

However, from someone in industry, there needs to be some form of repository to collect the what-seems-to-be-useless-now work. I've run into various situations thru the years where some abstract paper from a 1960 Russian paper has been priceless. Since we can't predict the future, the seemingly mundane can't be fully appraised.

Maybe limiting the for-profit groups and scaling up PLOS?

Permalink to Comment

3. RB Woodweird on June 24, 2010 8:40 AM writes...

This is predicted by The First Law of Metridynamics, which states:

The observed metric will improve.

So when the system told chemists that one of the most important metrics which would determine advancement was the number of publications, the number of publications was of course going to increase.

(The alert reader will recall also The Second Law of Metridynamics, which states:

The sum of all metrics in a closed system is a constant.)

Permalink to Comment

4. Greg Hlatky on June 24, 2010 8:50 AM writes...

"...redundant, inconsequential, and outright poor research..."

Hey, that's me you're talking about!

Permalink to Comment

5. fungus on June 24, 2010 8:58 AM writes...

Woodweird - there's a name for it, it's called "Goodhart's Law".

As soon as something is measured, it changes.

http://en.wikipedia.org/wiki/Goodhart's_law

Permalink to Comment

6. dWj on June 24, 2010 9:34 AM writes...

Partly on Duct Tape's comment, I think a lot of these results would be of some value if stuck as a single entry in a table in the CRC book. The problem is one of organization of results, such that some theorist who thinks that seeing as many known ways to "dehydrate a primary amide to a nitrile" as possible will help construct some theory of molecular dynamics can find that in a useful form, while people to whom it's of no value don't have to sort through it. The ontological studies that determine the enthalpy of formation of 2-methyl-pentane are less highly regarded than they once were, and don't make a great scientist, but they can be worth their cost if performed by someone who isn't going to be publishing in Science anyway. These things don't need to be taking up space in great journals -- they maybe shouldn't all be in Physica A -- but they should be collected somewhere. What we need isn't fewer publications; what we need is a librarian.

Permalink to Comment

7. Tok on June 24, 2010 9:38 AM writes...

Anon - I'd feel sorry for the first grad students through your system and their 10-15 year degree time. We're already at what 6 years now?

Permalink to Comment

8. Anon on June 24, 2010 10:18 AM writes...

Re:Tok
I think you have misunderstood what I meant. Perhaps I should have phrased my thoughts better. I'm not arguing that we should keep students longer (in fact I believe anything longer than 5 yrs for a PhD is way too long). You shouldn't have to feel bad for my non-existent students b/c if we eliminated the "who has the most publications" game then grad students don't need 10-15 yrs. They do the normal 5 yrs and go on their merry way with the science they have accomplished, and we judge them based on that science. Then when job time comes they will present their research to the hiring committee and stand or fall based on their science rather than on some silly number (of publications). We've all seen people that pad themselves up with meaningless publications. How are people that don't pad themselves supposed to compete if the game is set up from the beginning to favor those with