Corante

About this Author
DBL%20Hendrix%20small.png College chemistry, 1983

Derek Lowe The 2002 Model

Dbl%20new%20portrait%20B%26W.png After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases. To contact Derek email him directly: derekb.lowe@gmail.com Twitter: Dereklowe

Chemistry and Drug Data: Drugbank
Emolecules
ChemSpider
Chempedia Lab
Synthetic Pages
Organic Chemistry Portal
PubChem
Not Voodoo
DailyMed
Druglib
Clinicaltrials.gov

Chemistry and Pharma Blogs:
Org Prep Daily
The Haystack
Kilomentor
A New Merck, Reviewed
Liberal Arts Chemistry
Electron Pusher
All Things Metathesis
C&E News Blogs
Chemiotics II
Chemical Space
Noel O'Blog
In Vivo Blog
Terra Sigilatta
BBSRC/Douglas Kell
ChemBark
Realizations in Biostatistics
Chemjobber
Pharmalot
ChemSpider Blog
Pharmagossip
Med-Chemist
Organic Chem - Education & Industry
Pharma Strategy Blog
No Name No Slogan
Practical Fragments
SimBioSys
The Curious Wavefunction
Natural Product Man
Fragment Literature
Chemistry World Blog
Synthetic Nature
Chemistry Blog
Synthesizing Ideas
Business|Bytes|Genes|Molecules
Eye on FDA
Chemical Forums
Depth-First
Symyx Blog
Sceptical Chymist
Lamentations on Chemistry
Computational Organic Chemistry
Mining Drugs
Henry Rzepa


Science Blogs and News:
Bad Science
The Loom
Uncertain Principles
Fierce Biotech
Blogs for Industry
Omics! Omics!
Young Female Scientist
Notional Slurry
Nobel Intent
SciTech Daily
Science Blog
FuturePundit
Aetiology
Gene Expression (I)
Gene Expression (II)
Sciencebase
Pharyngula
Adventures in Ethics and Science
Transterrestrial Musings
Slashdot Science
Cosmic Variance
Biology News Net


Medical Blogs
DB's Medical Rants
Science-Based Medicine
GruntDoc
Respectful Insolence
Diabetes Mine


Economics and Business
Marginal Revolution
The Volokh Conspiracy
Knowledge Problem


Politics / Current Events
Virginia Postrel
Instapundit
Belmont Club
Mickey Kaus


Belles Lettres
Uncouth Reflections
Arts and Letters Daily
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline

In the Pipeline

« 500,000 Excess Deaths From Vioxx? Where? | Main | Return of the Rhodanome »

May 31, 2012

Anonymous Grant Review Gets A Try

Email This Entry

Posted by Derek

Anonymity is a topic that comes up whenever you talk about commenting on published scientific work. Some people are very uncomfortable with the idea of others being able to take potshots at them from behind convenient rocks, while others think that without that ability, a lot of relevant discussion will never take place.

Similar concerns apply to academic research grants. A big name never hurts - but what if all the names were stripped off the proposals? Many people have wondered this over the years, but now the NSF has been giving it a try:

Known as The Big Pitch and launched 2 years ago by officials in the agency's Molecular and Cellular Biosciences (MCB) Division, the effort aims to find out if making proposals anonymous—and shorter—has an impact on how they fare in the review process. “We wanted to find ways to identify transformative ideas that are getting lost in the regular peer-review process,” says Parag Chitnis, head of the MCB division. “So we asked: What would happen if we strip off the name of the PI [principal investigator] and institution and distill proposals down to just the big question or the core idea?”

What happens is a lot, according to the first two rounds of the Big Pitch. NSF's grant reviewers who evaluated short, anonymized proposals picked a largely different set of projects to fund compared with those chosen by reviewers presented with standard, full-length versions of the same proposals.

They're tried this twice, in two different research areas, each time with some 50 to 60 proposals to work with. Both times, the full-proposal rankings were almost completely different than the anonymous-pitch ones. I can see some problems with drawing conclusions here, though: for one thing, if two different teams of evaluators look over the same set of proposals (in either format), how closely do they agree? I'd like to see the NSF try that experiment - say, three different panels rating each set. And I'd include a third group, the condensed proposals with the names still on them. That might help answer several questions: how much do such panels diverge in general? Is the spread larger or smaller with the condensed proposal format? With the names stripped off? How much of the difference in rating is due to each factor?

These ideas have occurred to the people involved, naturally:

The experiment was not designed to separate out the effect of anonymity, but it may have been a factor. In both Big Pitch rounds, reviewers evaluating the anonymous two-pagers were later told the identity of the applicants. In some cases, Chitnis says, panelists were surprised to learn that a highly rated two-pager had come from a researcher they had never heard of. In others, he notes, reviewers “thought they knew who this person is going to be” only to find that the application came from a former student of the presumed bigwig, working at a small institution.

In their next round, the NSF plans to try to sort some of these factors out. I very much hope that this sort of thing continues, though. There should be a mixture of funding mechanisms out there: programs that fund interesting people, no matter what they're working on, and ones that fund interesting ideas, no matter where they came from.

Comments (26) + TrackBacks (0) | Category: Academia (vs. Industry)


COMMENTS

1. mass_speccer on May 31, 2012 8:21 AM writes...

Another problem with this data is the question of which works better - the different sets of proposals got different outcomes, but maybe the short proposals led to funding of poor projects? (or vice versa of course).

Also, I've always been told to avoid applying for funding for projects that look similar to my former supervisors work - doesn't show much originality. Do funding bodies actually take this into account? (It would obviously be rather difficult with anonymous applications).

I guess it would be difficult to write these, no more "we have previously shown that...."!

Permalink to Comment

2. Brian on May 31, 2012 8:34 AM writes...

The NIH has done one of these control experiments - shortening the application. The research proposal part of most applications was cut in half a year or two (or three?) ago. Opinions I have heard think that this favors more experienced investigators, since they can rely on publications to make up for the loss of space, which limits the amount of preliminary data that can be presented in the context of a good story.

Permalink to Comment

3. Wile E. Coyote, Genius on May 31, 2012 8:47 AM writes...

Derek,

I disagree with your closing statement "There should be a mixture of funding mechanisms out there: programs that fund interesting people, no matter what they're working on, and ones that fund interesting ideas, no matter where they came from."

There are many interesting people out there. I don't think that simply being "interesting" deserves funding with tax payers' dollars. Instead, I'd prefer to see my tax money going to those researchers with good, novel, groundbreaking proposals, no matter where they come from.

Permalink to Comment

4. DLIB on May 31, 2012 9:30 AM writes...

Agree with three.

Glad to see this and I hope this becomes standard. I submitted a proposal to the NIH once and received 1's and 2's ( some differences between reviewers ) when it came to the idea and approach and when it came to the PI I consistently got the lowest score - I'm a nobody.

Permalink to Comment

5. Helical_Investor on May 31, 2012 9:53 AM writes...

So ... when submitting a grant, reference a lot of 'related' prior work from a big name researcher lab so as to give the impression that one is that researcher.

Or am I too cynical?

Permalink to Comment

6. entropyGain on May 31, 2012 10:00 AM writes...

Good ideas are everywhere. The ability to execute them is a bit more rare. Don't you think trackrecord environment should contribute to the most informed decision about project funding? Yes, sometimes high profile types try to get by on their reputation alone, and some fool reviewers may buy it, but most of the times I've seen that in a Study Section, the reviewers see through it. There is also a considerable reluctance to give additional funds to an already well funded lab.

And since there are so many med chemists on this blog: how many times have you heard people propose really creative/interesting molecules but they don't have the hands to make it? The chemist who spouts off "creative" ideas in meeting, but never seems to register breakthrough molecules shouldn't get funded....

Permalink to Comment

7. David Formerly Known as a Chemist on May 31, 2012 10:07 AM writes...

The grant review system is an insider's paradise, just like the stock market. I know academic researchers who are well aware of who will be sitting on study panels, ensure their applications get into study panels stacked with "friends", and then seem to know how well their review went the same day of the review meeting (even though scores and pink sheets aren't sent out until weeks later). Like any mixture of humans and money, people find ways to game the system. Dirty? Reality.

Permalink to Comment

8. DLIB on May 31, 2012 10:27 AM writes...

The resource question pops up regarding this.

The review panel simply reviews the approach/ novelty...of the proposal anonymously and gives their rankings. Subsequently, the program manager can determine if the PI ( unmasked ) has the resources to execute the project.

This is technically a contract between the government and the institution. If the government feels that the institution has no means to execute the contract then it should not enter into the contract.

Permalink to Comment

9. Hap on May 31, 2012 11:20 AM writes...

Is it easier to judge from a longer proposal or a shorter proposal whether the people implementing it have a neat useful idea or whether they're the lab equivalent of the "idea rat"?

Experience should count for something, but I'm not sure whether the experience with well-known people is "being connected enough to have received enough money to do something useful" or "being useful".

Permalink to Comment

10. CR on May 31, 2012 11:31 AM writes...

To #8, DLIB:

I agree with this to some extent. The ability to carry out the research is a big question when I review grants. And your idea does help with this; however, what happens after the review and the science offer deems that 40% of the grants cannot be executed due to lack of resources? Do you reconvene the committee to re-review the grants? Or do you just keep going down the line until you hit the magic number? I think the ability to carry out the proposal is part of the review and should be factored in when giving a score, not simply the novelty of the grant. I think reviewing manuscripts in an anonymous fashion is an excellent idea - and some journals send them out without revealing the authors; but with grants, knowing the institution/PI's background/resources is an intimate part of the proposal.

Permalink to Comment

11. Student on May 31, 2012 11:31 AM writes...

@6 "The ability to execute them is a bit more rare"
Maybe those are just the investigators/personalities that tell their grad students/postdocs what the data should look like...and how much that data might help their career...Maybe I'm a cynic..but I'm sure Nature will put out a piece on this issue sometime in the future.

Permalink to Comment

12. The Iron Chemist on May 31, 2012 12:19 PM writes...

Instead of "The ability to execute them is a bit more rare," it might be "The opportunity to execute them is a bit more rare." There are plenty of highly competent people at lower-ranked institutions. Seemingly, the talent gap between those who land the plum academic positions and those who lose out entirely has never been narrower.

Permalink to Comment

13. MTK on May 31, 2012 12:48 PM writes...

A variety of funding mechanisms would be a good thing IMO. Sort of a simple way to ensure diversification which is always good when managing a portfolio.

At a minimum, separating the review of the idea and the ability to execute on that idea is probably not a bad idea either.

Permalink to Comment

14. biotechchap on May 31, 2012 1:49 PM writes...

You always target big pharma , Please also comment on the following :


Vertex Pharmaceuticals (MA) (VRTX) Goes From Bad Data to a Bad Promotion


5/31/2012 6:58:45 AM

This is not a good week for the executive team at Vertex Pharmaceuticals. Two days ago, the drugmaker revised previously announced interim results of a Phase II study of a combination cystic fibrosis treatment, taking the wind out of its once hot stock and tarnishing its credibility (see here). Now, the FDA has issued a so-called untitled letter for a promotional push for its Incivek hepatitis C treatment for including misleading material.

Permalink to Comment

15. Anonymous on May 31, 2012 3:33 PM writes...

@7, I totally agree and have also witnessed it in my academic life.

Permalink to Comment

16. DLIB on May 31, 2012 6:49 PM writes...

@CR:

I think what should happen is based on the circumstance...by the time the initial hurdle is passed there would be less proposals to deal with so this type of attention is possible. There are obviously different notions of the amount / type of resources are required to answer a scientific question. The first and easiest possibility to deal with is that the program manager does not believe the PI asked for enough money to carry out the research. In that scenario they should discuss the approach and it's realism. On grants that are not the highest in the group it is often the case less money is granted after discussions with the PI. A second possibility is that the proper facilities are not resident or accessible to the PI. The program director should talk to them to see if there is a plan to address this. A reason that should have the least amount of weight is that they are not well known in the field - the point of the anonymity in the first place. If it is deemed the PI has no plan or a completely unbelievable plan to execute his ideas and approaches that otherwise scientifically are highly meritorious ( as the panel finds )then money should not be given to that PI - it is not a law of nature that all allocated money has to be spent ( although I've seen people spend quickly on a gross of sharpies to justify the amounts they ask for). It is also possible that a collaboration with a PI the Program Manager suggests to help shepherd the project could be a means to get important work done.

Permalink to Comment

17. KCN on May 31, 2012 9:46 PM writes...

As long as I get the funding to synthesize rationally designed analogs of maitotoxin for potential therapeutic leads, I'm fine with either scenario.

Permalink to Comment

18. Vanad on June 1, 2012 6:54 AM writes...

The iron Chemist: you are dead right....

and?

Permalink to Comment

19. anonyMouse on June 1, 2012 7:58 AM writes...

OTOH, my advisor, who has a big name, writes proposals of highly variable quality. The lousy proposals are funded just about as frequently as the good ones, presumably because of his name rather than the quality of the ideas/practicality of execution of the proposal.

Permalink to Comment

20. Liz on June 1, 2012 9:33 AM writes...

I would strongly favor 'great ideas' over 'interesting people'. Sure, the interesting people likely have the research organization under them to carry out the proposed research successfully, but proposing great ideas then funding them leads to further advancement of the science of the times.

This is not to say a great idea today is a great idea tomorrow, next year or in the next decade. We need only review Farber's idea to test folic acid in leukemia to see that it was a great idea then, but would not be so now.

Permalink to Comment

21. The Iron Chemist on June 1, 2012 10:24 AM writes...

@18: You're right, I didn't fully finish the thought.

I would add that a lot of the PIs from middle-tier schools are getting hosed for no good reason. These are people who have proven that they can design and carry out high-quality research but are getting dinged because they didn't win the top prize in the lottery the year they went on the job market. You can have essentially the same guy, with adequate resources for the project, get a 1 at one institution and a 4 at another.

Permalink to Comment

22. Old Timer on June 2, 2012 6:18 AM writes...

Iron Chemist: I think your understanding of history needs correcting. There has always been a narrow gap "between those who land the plum academic positions and those who lose out entirely." YOU are not judged for being in a "plum academic position" or not, it's your group and resources that are judged. There is a significant gap between the students/postdocs available at the "plum" institutions and those even a little lower down (think top 5 vs top 20), unfortunately.

Permalink to Comment

23. Ivy Sham on June 2, 2012 11:54 AM writes...

Old timer: You could not be more wrong. The defining characteristic of a graduate student at a top 5 school is that their parents wrote huge checks to the student's undergrad institution. Then, the top 5 schools take only kids from similar tier schools. As a former postdoc at an ultra-elite ivy institution, I can assure you that most of the faculty are not worth crap, having gotten their plush jobs as a result of being the favorite son of some other over-the-hill faculty member. Take a stroll through the recent literature and you will see that a significant majority of them are working on problems of the day circa-1990. It is well known that many members of NIH panels grant money to these has-beens simply based on their legacy and not their ability to address the significant issues of the day.

Permalink to Comment

24. Old Timer on June 3, 2012 6:33 AM writes...

My mistake.

Permalink to Comment

25. Ted Dibble on June 3, 2012 9:55 PM writes...

@23 My (Chemistry) department at the small and state school where I work recently placed a B.S. degree recipient in the graduate program at MIT and another at Yale. These students did not get there by having parents write big checks!

@21 I suspect the discrepancy is usually much smaller than what you suggest, but also know it does not take much of a difference to make a proposal unfundable. On the other hand, I have been on a couple NSF panels and seen proposals from well-known people and/or institutions get very low scores because the proposals were poorly written or otherwise flawed.

Permalink to Comment

26. CR on June 5, 2012 12:10 PM writes...

@#23, IvySham...

Bitter much? Sorry you didn't get that job you wanted...better luck next time.

Permalink to Comment

POST A COMMENT




Remember Me?



EMAIL THIS ENTRY TO A FRIEND

Email this entry to:

Your email address:

Message (optional):




RELATED ENTRIES
One and Done
The Latest Protein-Protein Compounds
Professor Fukuyama's Solvent Peaks
Novartis Gets Out of RNAi
Total Synthesis in Flow
Sweet Reason Lands On Its Face
More on the Science Chemogenomic Signatures Paper
Biology Maybe Right, Chemistry Ridiculously Wrong