« SRT1720: Good (And Confusing) News for Obese Mice |
Main
| Disappearing Information, Courtesy of Aldrich Chemical »
August 24, 2011
What Are Impact Factors Doing to Chinese Science?
Posted by Derek
Here's a useful (and rather brave) editorial from Chinese chemist Nai-Xing Wang in Nature. He's pointing out that the government's funding agencies are taking a crude and harmful approach to who gets research support: publish, and publish according to their schemes, or flippin' well perish:
The biggest problem remains the obsession with journal impact factors. Generally speaking, articles in journals with high impact factors are judged to appeal most to readers, but not every paper published in a high-impact-factor journal is high quality, and papers published in lower-ranked journals are never worthless. Yet some administrators in China take a very crude approach: high-impact-factor publications mean excellent work.
Research proposals are judged according to the impact factor of a scientist's previous publications. (And referees are usually selected on these criteria too.) Worse, the salaries of my chemistry colleagues go up or down depending on a complex mathematical formula based on the impact factor of the journals in which we publish our work — which we must supply in a detailed list.
Now, this sort of thing has been going on around the scientific world for a while, and it's going to be hard to put a complete stop to it. But the Chinese system that's described is about the most blatant that I've come across. The effects are pernicious:
If a high impact factor is the only goal of chemistry research, then chemistry is no longer science. It is changed to a field of fame and game. There are other effects too. Administrators in almost every university and research institute like to evaluate researchers by their papers at the end of each year. As a result, chemists often choose easy research topics that can be written up inside a year. . .
You get what you subsidize; I don't think that law is ever broken. And if the Chinese government wants people to crank out lots of papers in what they feel are high-end journals, well, that's what people will do. But if they want something useful to come out of all that effort, well, things might need to be adjusted a bit. But "useful" is a slippery word. For the people who are gainfully employed in keeping the current system running, it's just about as useful as it can be the way it is.
Comments (26)
+ TrackBacks (0) | Category: The Scientific Literature
POST A COMMENT
- RELATED ENTRIES
- › Roger Boisjoly and the Management Hat
- › Every Methods Paper Has a Table
- › Buying Back Shares: An Admission of Defeat
- › More Industrial Espionage
- › Tau Spreads On Its Own?
- › Academia and Industry, Suing Each Other
- › Let's Start Off the Meeting With An Ad, OK?
- › The Academic-Industrial Collaboration in Drug Discovery Panel: Today
1. Another student on August 24, 2011 9:53 AM writes...
Would it be a stretch to compare this to teaching kids to take standardized tests? You don't learn the material but you learn the game. I can actually see this leading to some nasty scientific-politics and the formation of unspoken alliances to generate the maximum benefit.
Permalink to Comment2. CMCguy on August 24, 2011 10:08 AM writes...
Gosh I only thought the Chinese had sent their Spies to the infiltrate the labs and technical areas but apparently there must have been a few in Pharma Management to learn the concepts of "research by the numbers" and "only count blockbusters". Although not really sure the NIH "old boys network" of funding distribution has done much to accelerate advancements either. Problems occur when want to avoid or over control risks when it's taking risks that are most likely provide true creative innovation.
Permalink to Comment3. anchor on August 24, 2011 10:10 AM writes...
Permalink to CommentOnce corrupt..always corrupt! Ms. Nai-Xing Wang's open expression is very brave and we all hope that when all is said and done, she still can function effectively and still be a productive chemist.
4. MTK on August 24, 2011 10:15 AM writes...
I'm not sure how brave it is.
She got something published in Nature; a journal with a high impact score.
It'll look good for her next performance review.
Permalink to Comment5. NoDrugsNoJobs on August 24, 2011 10:16 AM writes...
I reckon they are being explicit about what is already implicitly true here - Makes them more upfront about what the criteria are rather than obfuscating
Permalink to Comment6. Ash on August 24, 2011 10:24 AM writes...
Seems to be consistent with the rumor that Chinese scientists are promised tens of thousands (and sometimes a hundred thousand) dollars if they publish in Nature or Science.
Permalink to Comment7. Anatoly on August 24, 2011 10:34 AM writes...
Well, as long as Journal's Impact Number is based on citations by other scientists, and citations are honest, the system should work.
But I wonder if Nature only publishes articles that cite other articles from Nature. Or that perhaps the high impact journals form a cartel where they only publish those who cite from a pool of their own articles...
Permalink to Comment8. Innovorich on August 24, 2011 10:42 AM writes...
Errrm - isn't this how it works in the US too??
Permalink to Comment9. RB Woodweird on August 24, 2011 10:57 AM writes...
"You get what you subsidize; I don't think that law is ever broken."
It is a restatement of the First Law of Metridynamics: The observed metric will improve.
Permalink to Comment10. MTK on August 24, 2011 11:39 AM writes...
Well, the whole thing is gaming isn't it.
The journals game the system to get as high an impact score as possible. We game the system to get our papers into high scoring journals which leads to better grant reviews, etc., etc.
I guess that's what happens anytime you have a bureaucracy. And by that I mean any organization over five people.
Permalink to Comment11. David Formerly Known as a Chemist on August 24, 2011 12:00 PM writes...
This isn't at all shocking. When someone knows what factor is being measured, that factor improves. Remember 10 years ago, when pharma started measuring the productivity of chemists based on the number of new compounds they registered? We saw an explosion of new compounds (aided by combichem and parallel synthesis), but how much value was provided by those new compounds? Very little, if the flushing and pruning of corporate collections over the past 5 years are any indication.
Permalink to Comment12. simpl on August 24, 2011 12:00 PM writes...
The people working at our chemical development and production plant in China have impressed me (I have no contact with the research team, alas). Maybe they see interesting and useful work as an viable alternative?
Permalink to Comment13. Henning Makholm on August 24, 2011 12:01 PM writes...
@Innovorich, it's how it works everywhere. The remarkable thing is that it's being said _in China_, since the Chinese government is known for being somewhat more ... enthusiastic ... about stifling dissent than many other governments.
(Which is to be understood relatively: if you criticize a bad policy, those responsible for that policy are always going to find a way to get back at you, no matter where you do it. The difference is that in an authoritarian system there are going to be people in power who _despite_ not being personally threatened by your criticism but will join in the getting-back on general principles, because the system views any dissent as inherently suspect).
Permalink to Comment14. RD on August 24, 2011 12:21 PM writes...
Welcome to America!
Permalink to CommentAnd you thought we would have to compete against the Chinese. Now even they can indulge in petty, research killing politics, corralling resources for their next publications and kissing the asses of the people two levels above them while stabbing the backs of their colleagues.
The values of the ultracompetitive business class trickle down once again. It's all hierarchy, status and conformity.
What we need is a good plague.
15. MoMo on August 24, 2011 1:46 PM writes...
Impact factors of journals are useless tripe. They should be studying the citation analysis and number of times cited and whether the paper generates a scientific front via cluster analysis techniques- pioneered by Eugene Garfield at ISI now Thompson.
That is how you know science is valuable, and the smart people use this. This keeps the Chinese in the dumb-ass category relegated to others like those Pakistani publishers scientists love to get their names attached to.
China, if you are listening, now is the time to avoid mistakes aand create the uber-master scientific race!
But first, who wants to hear a story why you shouldnt outsource to China? Its a double knee-slapper!
Permalink to Comment16. Anonymous on August 24, 2011 2:25 PM writes...
Henning Makholm, we should begin placing bets on whether Wang will "mysteriously" disappear.
Permalink to Comment17. Pete on August 24, 2011 4:56 PM writes...
Could impact factor be a Trojan Horse? On the subject of useless metrics, was there any mention of h-indices? Perhaps the journals could find a way to take a percentage of the prize money.
Permalink to Comment18. K. Marx on August 24, 2011 5:16 PM writes...
Pure Capitalism!!!
Permalink to Comment19. Jimmy on August 24, 2011 8:10 PM writes...
Regarding the "useful" research comment, the number of patents coming out of China shows that that huge amount of research isn't resulting in commercial gain. That said it may be more of a priority to prove that China has "good" research, somewhat akin to the cold war Olympics.
Permalink to Comment20. anonymous on August 24, 2011 10:08 PM writes...
this equal that.... how much time have you search for a publication and finding a dark chinese journal giving info on what you want to do...
you try it, thinking : hey it's been published' and finally another turd at the bottom of the line...
yeah! right! there's something fishy when there is too much vowels in the journal name :p
Permalink to Comment21. Ich Dich on August 25, 2011 1:39 AM writes...
This reminds me of the expedition of the Nobel committee to China (I think it was 2006, but I could be wrong) where they had to explain how Chinese could win Nobel prizes.
Permalink to CommentWhat would the financial reward be if you would win a Nobel prize in China? Perhaps higher than the actual prize...
22. Anonymous on August 25, 2011 7:27 AM writes...
So does anyone have an idea of a surrogate measure of scientific quality? The mentioned h-factor doesn't seem to get popular for some reason, does it?
Permalink to Comment23. Artmisinin on August 25, 2011 8:41 AM writes...
Last year, two biologists criticized China's funding system in Science (Y Shi and Y Rao, Science, 2010, 329, 1128 (DOI: 10.1126/science.1196916). I totally agree with them that the problem is about the system and culture.
China does not have a working peer review system to ensure merit based funding. To make the funding mechanism relatively fair, there must be something that can measure applicants' performance by numbers. Otherwise, things will be more ugly.
Another similar situation is China's college entrance examination - all the universities pick students according to their exam scores, nothing else. Why? Just try to leave no room for foul plays.
You American scientists should feel good about your system. It is not perfect, but it is working in a relatively efficient way to promote innovation. We see so many Chinese talents achieved much more here in the States than their peers in China. It certainly is not because that they are smarter or they work harder...
Permalink to Comment24. gyges on August 25, 2011 12:00 PM writes...
"the salaries of my chemistry colleagues go up or down depending on a complex mathematical formula based on the impact factor of the journals in which we publish our work "
I've seen post-doc positions advertised in the UK where one of the selection criteria to be used is the number of publications that each of the applicants have.
It looks as though we're going down this Road to Serfdom too.
Permalink to Comment25. CR on August 25, 2011 12:07 PM writes...
@Anon, #20:
"yeah! right! there's something fishy when there is too much vowels in the journal name :p"
Then I guess Tetrahedron Letters has too many vowels.
Permalink to Comment26. loupgarous on January 31, 2012 10:54 PM writes...
The Chinese are putting a rod in pickle for their own backs by making publication impact the overriding metric for evaluating the success or failure of individual researchers. They're going to have a whole culture built around prestige and success in publications, without the necessary freedom to question published results of prestigious authors - no real audit.
In "Carnage and Culture," historian Victor Davis Hanson observes that while ancient Chinese culture outstripped Western culture at the very beginning of the Enlightenment, the evolving process of audit caused the nations of the West to be able to prevail militarily and commercially over China and the other great Eastern cultures.
If Chinese scholarship comes to emulate Chinese crony capitalism, they can publish all they wish, with no audit process to give the scandals with which Western scholars are all too familiar - and at the end of the day truth will be the main casualty.
Finding the truth is st times a messy process - you get Baltimore scandals, Duke biomarker scandals, all sorts of unpleasantness. Perhaps the main determinant of a culture's success is its willingness to tolerate the mess which leads to the truth.
A face-saving culture seems to be handicapped in this way; if it can't tolerate Kuhnian paradigm changes and exposure of academic misconduct, then it will proceed with dignity along the path of error while more flexible cultures cleave closer to the truth.
Permalink to Comment