Anonymity is a topic that comes up whenever you talk about commenting on published scientific work. Some people are very uncomfortable with the idea of others being able to take potshots at them from behind convenient rocks, while others think that without that ability, a lot of relevant discussion will never take place.
Similar concerns apply to academic research grants. A big name never hurts - but what if all the names were stripped off the proposals? Many people have wondered this over the years, but now the NSF has been giving it a try:
Known as The Big Pitch and launched 2 years ago by officials in the agency's Molecular and Cellular Biosciences (MCB) Division, the effort aims to find out if making proposals anonymous—and shorter—has an impact on how they fare in the review process. “We wanted to find ways to identify transformative ideas that are getting lost in the regular peer-review process,” says Parag Chitnis, head of the MCB division. “So we asked: What would happen if we strip off the name of the PI [principal investigator] and institution and distill proposals down to just the big question or the core idea?”
What happens is a lot, according to the first two rounds of the Big Pitch. NSF's grant reviewers who evaluated short, anonymized proposals picked a largely different set of projects to fund compared with those chosen by reviewers presented with standard, full-length versions of the same proposals.
They're tried this twice, in two different research areas, each time with some 50 to 60 proposals to work with. Both times, the full-proposal rankings were almost completely different than the anonymous-pitch ones. I can see some problems with drawing conclusions here, though: for one thing, if two different teams of evaluators look over the same set of proposals (in either format), how closely do they agree? I'd like to see the NSF try that experiment - say, three different panels rating each set. And I'd include a third group, the condensed proposals with the names still on them. That might help answer several questions: how much do such panels diverge in general? Is the spread larger or smaller with the condensed proposal format? With the names stripped off? How much of the difference in rating is due to each factor?
These ideas have occurred to the people involved, naturally:
The experiment was not designed to separate out the effect of anonymity, but it may have been a factor. In both Big Pitch rounds, reviewers evaluating the anonymous two-pagers were later told the identity of the applicants. In some cases, Chitnis says, panelists were surprised to learn that a highly rated two-pager had come from a researcher they had never heard of. In others, he notes, reviewers “thought they knew who this person is going to be” only to find that the application came from a former student of the presumed bigwig, working at a small institution.
In their next round, the NSF plans to try to sort some of these factors out. I very much hope that this sort of thing continues, though. There should be a mixture of funding mechanisms out there: programs that fund interesting people, no matter what they're working on, and ones that fund interesting ideas, no matter where they came from.