Corante

About this Author
DBL%20Hendrix%20small.png College chemistry, 1983

Derek Lowe The 2002 Model

Dbl%20new%20portrait%20B%26W.png After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases. To contact Derek email him directly: derekb.lowe@gmail.com Twitter: Dereklowe

Chemistry and Drug Data: Drugbank
Emolecules
ChemSpider
Chempedia Lab
Synthetic Pages
Organic Chemistry Portal
PubChem
Not Voodoo
DailyMed
Druglib
Clinicaltrials.gov

Chemistry and Pharma Blogs:
Org Prep Daily
The Haystack
Kilomentor
A New Merck, Reviewed
Liberal Arts Chemistry
Electron Pusher
All Things Metathesis
C&E News Blogs
Chemiotics II
Chemical Space
Noel O'Blog
In Vivo Blog
Terra Sigilatta
BBSRC/Douglas Kell
ChemBark
Realizations in Biostatistics
Chemjobber
Pharmalot
ChemSpider Blog
Pharmagossip
Med-Chemist
Organic Chem - Education & Industry
Pharma Strategy Blog
No Name No Slogan
Practical Fragments
SimBioSys
The Curious Wavefunction
Natural Product Man
Fragment Literature
Chemistry World Blog
Synthetic Nature
Chemistry Blog
Synthesizing Ideas
Business|Bytes|Genes|Molecules
Eye on FDA
Chemical Forums
Depth-First
Symyx Blog
Sceptical Chymist
Lamentations on Chemistry
Computational Organic Chemistry
Mining Drugs
Henry Rzepa


Science Blogs and News:
Bad Science
The Loom
Uncertain Principles
Fierce Biotech
Blogs for Industry
Omics! Omics!
Young Female Scientist
Notional Slurry
Nobel Intent
SciTech Daily
Science Blog
FuturePundit
Aetiology
Gene Expression (I)
Gene Expression (II)
Sciencebase
Pharyngula
Adventures in Ethics and Science
Transterrestrial Musings
Slashdot Science
Cosmic Variance
Biology News Net


Medical Blogs
DB's Medical Rants
Science-Based Medicine
GruntDoc
Respectful Insolence
Diabetes Mine


Economics and Business
Marginal Revolution
The Volokh Conspiracy
Knowledge Problem


Politics / Current Events
Virginia Postrel
Instapundit
Belmont Club
Mickey Kaus


Belles Lettres
Uncouth Reflections
Arts and Letters Daily
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline

In the Pipeline

« Here Be Chiral Dragons, With Fluorinated Fangs | Main | Maybe Not Improved, But Definitely New »

November 14, 2007

How You Doin'? How's Everybody Doin'?

Email This Entry

Posted by Derek

At many companies, this is performance review season. As I’ve written about before, this is a particularly hard thing to do right in a research organization. It’s so hard, actually, that never once have I heard of one where the scientists were satisfied with how people were being rated. I think it’s probably impossible for any organization, if you want to know the truth. It’s like trying to design a perfect voting system. No matter what happens, some people are going to feel, perhaps even with justification, as if they’ve been had.

But evaluating scientists is especially thankless. If you have a lot of really good ones, it’s a little like filling out yearly reports on poets. Hmmm. . . Mr. Larkin. I see you haven’t published anything so far this year, and still no collection since The Whitsun Weddings. . .wasn’t that on your goals statement for this period? I don’t really see how we can give you an “exceeds” rating given all that. And Mr. Lowell, it’s true that you produced a great number of sonnets during this review period, but I can’t help but believe that these were less of an effort than some of the work you’ve done for us before, and they certainly had less of an impact on our operations. No, I think that “meets expectations” is probably the correct category this year. . .And as for you, Mr. Housman, we need to ask ourselves just how long it has been since A Shropshire Lad. . .

Rating research productivity sends you into the same thickets. If someone hammered out a long list of analogs, but used pretty much the same chemistry to make each of them, how do you rate that compared to someone who had to hand-forge everything (and produced a correspondingly smaller pile)? How much should number of compounds count for, anyway – how about impact? What if the big bunch of compounds didn’t do much for the project, but one of the tough ones opened up a whole new area? (Or what if it was the reverse?) But isn’t that partly luck – what if the one that hit was totally unexpected, even by the person who made it? What if it became a great compound for reasons totally out of their hands?

And then you get to the people who aren’t necessarily cranking out analogs, the lab heads and such. They’re supposed to be leading projects, managing direct reports, coming up with ideas. How’d they do? How can you tell? Can you reliably distinguish a project that got lucky, or had a better starting point, from a well-managed one that has nonetheless been wandering around in the wilderness? Put your best people on, say, a protein-DNA interaction target, and pretty soon they won’t look so good, either.

No, even with the best rating system in the world, it would be hard to fill out the reports on drug discovery projects. And you can take it as given that no one is using the best rating system in the world. (Some may in fact be experimenting with the worst). The yearly frequency of ratings is one problem – anything tied to the calendar is a potential problem, since the compounds, the cells, and the rats never know what month it is. This has been a problem for a long, long time. I once quoted from Rayleigh’s biography of physicist J. J. Thomson. You wouldn’t want to run a whole department on the following system, but you don’t want to ignore the man’s point, either:

"If you pay a man a salary for doing research, he and you will want to have something to point to at the end of the year to show that the money has not been wasted. In promising work of the highest class, however, results do not come in this regular fashion., in fact years may pass without any tangible results being obtained, and the position of the paid worker would be very embarrassing and he would naturally take to work on a lower, or at any rate, different plane where he could be sure of getting year by year tangible results which would justify his salary. The position is this: you want this kind of research, but if you pay a man to do it, it will drive him to research of a different kind. The only thing to do is to pay him for doing something else and give him enough leisure to do research for the love of it."

And the insistence of many HR departments that the ratings fall on a normal distribution is another problem. Sure, if you hired a few thousand random people and turned them loose on the work, you could expect some sort of bell curve, assuming that you’ve solved that problem of fairly evaluating them. But you didn’t hire your people at random, did you? Everyone’s supposed to be at some level of competence right from the start. Some of those performance distribution curves are reflecting the randomness of research or the defects in rating it, rather than any underlying truths about performance.

Comments (17) + TrackBacks (0) | Category: Business and Markets | Life in the Drug Labs


COMMENTS

1. John Spevacek on November 14, 2007 9:42 AM writes...

My first thought is that maybe there should be a 5 year lag before a performance evaluation is done. After that delay, the real value of the work can be seen and (more) properly evaluated. Did the work lead to a dead end or did if allow others to build off it?

But then reality raises it's ugly head: will you even be working for the same company 5 years hence?

Permalink to Comment

2. Canuck Chemist on November 14, 2007 10:08 AM writes...

I think my last (relatively small) employer, with a team of 12-15 synthetic chemists, had it right. Salaries were standard for each level (they were quite generous). Bonuses were strictly based on team and company results, with the emphasis of people acting for the benefit of the team. They didn't want a busy person hoarding good ideas for themselves when someone else could help bring them to fruition. The team leader would give you subjective evaluations, but they were not fueled by e.g. HR-bell curves: the objective was only to make you a better scientist. This required them to be picky about who they brought in (no "lone wolfs" wanted), but all in all people were pretty happy. Even those who didn't quite see eye-to-eye with the team leader didn't take a hit to the pocketbook. Levels were also very well-defined, and you would get promoted as soon as you met the criteria required to move to the next level.
Sadly, an "approvable letter" by the FDA spelled the end of discovery research at this company...

Permalink to Comment

3. Sleepless in SSF on November 14, 2007 10:22 AM writes...

Derek: it seems like you might be missing the link to the Rayleigh quote. Or am I reading it wrong?

Permalink to Comment

4. weirdo on November 14, 2007 11:02 AM writes...

I agree this is probably the worst time of year for managers and scientists alike. Performance evaluations are incredibly tough in pharma -- the goals are concrete, but how (or whether!) one reaches them (or not) seems so random sometimes.

But the "bell curve" does serve a very useful purpose, if used correctly. It FORCES the various department heads to actually talk about the scientists with their colleagues, and back up their prejudices with facts. Yes, yes, yes, I've heard the "we only hire the top, so why do we need a curve?" argument before. But your "curve" is not across all scientists at all companies. It's for your scientists at your company. And, once you get above 30-40 scientists or so, you are going to have a bell-shaped curve, whether you want to or not, if you are truly being objective. Maybe 60% of your scientists would "exceed expectations" at Baby Biotech Y. But they "meet expectations" at Big Pharma X. And that's where they're being ranked.

(You can reverse those hypothetical rankings if you wish!)

I like the baseball team analogy, personally. You may be the starting shortstop for the Marlins, but you wouldn't be on the Yankees.


Like any tool, performance ranking systems can be mis-used. But, when used properly, I think the good old bell curve is pretty useful. We don't live in an egalitarian world; does anyone?

Permalink to Comment

5. milkshake on November 14, 2007 11:36 AM writes...

I think the bureaucratized system is very common in the industry. The argument usually goes about ensuring the career development and equal treatment of employees but it actually means "we dont trust the judgement of the lab bosses and we want to have a paper trail in case of a merit/performance-related argument or when some employee threatens to sue our company. So the formal rules are instituted for the HR benefit - to get a better scoop on whats happenning with the employees and provide amunition for defending the company and for keeping lid on the scientists salaries.

In my previous companies I usually wrote the evaluation on myself - completely uncritically exaggerated - and then asked my boss to put it in. Everybody was happier that way.

Permalink to Comment

6. Hap on November 14, 2007 11:59 AM writes...

Unless you've got a lot of people, a bell curve probably isn't likely - I don't think forty is enough to get one for sure. Bell curves work for lots of distributions, but there isn't a certainty that they'll work for this - if your HR dept. would be perfect, for example, there wouldn't be one (everyone would perform equally well), and while you were drawn from what may have been a pool following that curve (probably), it isn't necessarily true that a company's employees follow one (if the company's culture is screwed up, for example, no one may try to achieve, or only a few; as an alternative, you might also see bimodal distributions).

There is also the bonus that defining achievement and productivity is nontrivial, unless defined by obvious company-wide goals (which are grainy enough not to necessarily reward the appropriate people). Just because some quantity gives you a bell curve doesn't mean it's actually measuring anything (useful).

Permalink to Comment

7. SRC on November 14, 2007 12:44 PM writes...

Milkshake has hit the mark. The bottom line (pace accountants): statistics can't substitute for judgment.

Any system based upon a mechanical, algorithmic-type ("objective") evaluation so beloved of HR is easy to game. In academia, it's number of papers (thereby pushing down the LPU). In industry, it's number of compounds (i.e., a technician's evaluation).

Either measure is worthless - only the subjective judgment of someone with common sense is worth anything. But as Voltaire famously said, common sense is not common.

Permalink to Comment

8. MTK on November 14, 2007 1:24 PM writes...

Objective and quantifiable measures are a critical component of a performance review. Note that I said component. You have to have some data to justify your subjective assessment. The trick is to have a good mix of multiple objective measures and defendable subjective measures by which to grade performance and to be consistent at all times.

The other thing that often leads to consternation is that managers and managed often let things fester for way too long. If you have issues or see potential issues, do not wait until the end of the year. I had a boss set up regular quarterly mini-reviews, which I thought was one of the best things he did. No surprises by the end and most people thought they had been treated fairly.

Permalink to Comment

9. milkshake on November 14, 2007 4:31 PM writes...

MTK: Evne the most objective and quantifyably -devised performance review depends on who is defining the review parameters and who is making the final evaluation (of what is the most succesful project or greatest performance).

Joseph Stalin was fond of saying: ""Those who cast the votes decide nothing. Those who count the votes decide everything"

Permalink to Comment

10. srp on November 14, 2007 7:47 PM writes...

This is an eternal problem in a wide range of organizations and functional areas. Lots of jobs are hard to measure--think about a director of advertising, where it's almost impossible to causally link sales to ads. ("I know half of my advertising is wasted but I don't know which half.")

Those advocating the use of lab directors' judgment as the primary criterion for evaluation surprise me with their confidence in the fairness and competence of these individuals. Are these folks really that credible on average? If so, I'm impressed, because the commenter crowd here is pretty tough on anyone acting in a managerial capacity.

Finally, would some sort of anonymous peer evaluation system (the dreaded "360 degree evaluation") release otherwise impacted information about who is really doing good research work? Conceivably, co-workers with similar training and close proximity to working conditions on given projects might have a better clue than anyone else.

Permalink to Comment

11. milkshake on November 14, 2007 9:48 PM writes...

Lab bosses dont have to be perfetly fair to their people or absolutely objective in evaluating their work - as long as the group produces the results. Give the bosses a fixed budget, the approximate time frame (like 2 years to come up with a preclinical candidate) and let them do their job without interference. If they succeed its bonus time for everyone. If they don't the boss standing is reduced - he gets less money and people in the next project.

Its the boss responsibilty how he runs his group and research - if there is a management problem, micromanagement from outside wont help to fix it.
If you start second guessing the project boss, you might as well replace him with someone else - and preferable more qualified - not with the HR.

The only role I see for HR is handling the paperwork/company policies/benefits job. The interdepartmental disputes are best dealt with by the top management and the in-group- and research-related problems are too tricky for HR to handle and - draging out problems to HR often makes a troubled group even more loathsome

Permalink to Comment

12. Dave H on November 15, 2007 9:38 AM writes...

Oh.... this one is way too easy.

Weirdo,

The Marlins beat the Yankees last time they played each other in the World Series (2003).

Permalink to Comment

13. SRC on November 15, 2007 12:30 PM writes...

You have to have some data to justify your subjective assessment.

Why? Anything quantitative is easy to game, as mentioned previously. Let people vote with their feet, and change (or apply to change) groups. Free market and all that. Who is in demand? Who is not? (Whether or not anyone is actually allowed to move.)

The obvious objection is that more popular projects will attract more interest. OTOH, less popular projects provide a greater opportunity to stand out, thereby attracting contrarians.

Permalink to Comment

14. Phil-Z on November 15, 2007 12:49 PM writes...

It probably wouldn't work in this age of self-service NMR, Mass Spec, etc, but I used to occasionally suggest that the analytical service department rate the chemists. When I ran a mass spec lab I quickly learned who were the winners and who were the hacks. Some chemists would bring me lots of clean samples that were usually the expected targets. They would bring intermediates and side products and could usually suggest reasonable structures for the unkowns. Their samples were not contaminated with stopcock grease.
And then there were the hacks ....

Permalink to Comment

15. scigeeek on November 15, 2007 10:29 PM writes...

No matter how performance reviews are touted as objective, the truth is they are subjective as heck. Praise can be heaped on for nothing to make a n'eer do well look good or nitpicking can be done to good people who threaten their not so good managers. Also, there's the mandate to lower the overall rating, as per HR.
This is a hard time of the year in a very hard year. Layoffs abound-hidden ones too, such as Genentech this week-did you hear about it, or read about it in the news-neither did I, instead I heard from a saddened colleague. The official line is that Genentech does not do layoffs. It's a tough time for everyone-uh, except CEO's and other members of the board or executives committees.

Permalink to Comment

16. weirdo on November 16, 2007 12:20 AM writes...

Dave H:

Immaterial. Performance ratings should have nothing to do with cross-team (company) comparisons. They MUST deal with how individuals are performing at YOUR company. And I hate Pfizer (I mean the Yankees).

Hap:

As a manager, if you have three direct reports, you should be able to rank them 1-2-3. Certainly, if you have forty, you can at least put them into five bins (labelling those bins any way you see fit, high to low). There wouldn't necessarily be equal numbers in each bin (ringing that bell curve again!). You SHOULD be able to rank them 1-40. If one cannot, one should not be a manager of scientists.

Regarding the crack about HR, I personally would not work for a company that had that kind of influence over hiring decisions. Too, I hope you are kidding about the "everyone would perform equally well" comment. Again, personally, I do not see this as being possible but, more importantly, I find the mere idea itself unappetizing. The whole diversity thing, don't you know (a concept I find immensely appealing, even beautiful).

Permalink to Comment

17. Hap on November 16, 2007 2:12 PM writes...

MTK: I wasn't trying to crack HR - I figured that ideally, whoever is in charge of the hiring, you would want maximally productive employees (though I guess I don't know if there actually is a maximal productivity level). It's unlikely that they'll be the same (for lots of reasons), but what they do for a company might not be much different. Once you have employees with differing skills, even if (particularly if) they are evenly productive, you have subjective decisions on which is more valuable - there may not be a single ranking of employees on which you could get two reasonable managers to agree. I think generating a rank order of employee value is likely to be analogous to voting, where unless the circumstances are constrained (and thus perturbing the system strongly in their own way), a single rank ordering is unlikely to occur. Simple ranking also excludes the complexity of the differences between the values of employees near in ranking (when 1 is much more valuable than 2, or if 1 and 2 aren't much different in value).

I would figure that any manager can place employees into groups based on productivity, but most of the other judgements (value for example) are rather subjective. There is no guarantee (or even likelihood?) that the distribution in those bins will fit a bell distribution. The use of the bell distribution then fits the cost constraints of an employer but is no longer a rational assessment of value - hence employee frustration.

P.S. I don't think HR has that much choice in who to hire where I work, but they presumably select who our managers get to see and then set the salary which they will be offered if we want to hire them, which determines at least some of the set of employees we actually get.

Permalink to Comment

POST A COMMENT




Remember Me?



EMAIL THIS ENTRY TO A FRIEND

Email this entry to:

Your email address:

Message (optional):




RELATED ENTRIES
Scripps Update
What If Drug Patents Were Written Like Software Patents?
Stem Cells: The Center of "Right to Try"
Speaking of Polyphenols. . .
Dark Biology And Small Molecules
How Polyphenols Work, Perhaps?
More On Automated Medicinal Chemistry
Scripps Merging With USC?