Corante

About this Author
DBL%20Hendrix%20small.png College chemistry, 1983

Derek Lowe The 2002 Model

Dbl%20new%20portrait%20B%26W.png After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases. To contact Derek email him directly: derekb.lowe@gmail.com Twitter: Dereklowe

Chemistry and Drug Data: Drugbank
Emolecules
ChemSpider
Chempedia Lab
Synthetic Pages
Organic Chemistry Portal
PubChem
Not Voodoo
DailyMed
Druglib
Clinicaltrials.gov

Chemistry and Pharma Blogs:
Org Prep Daily
The Haystack
Kilomentor
A New Merck, Reviewed
Liberal Arts Chemistry
Electron Pusher
All Things Metathesis
C&E News Blogs
Chemiotics II
Chemical Space
Noel O'Blog
In Vivo Blog
Terra Sigilatta
BBSRC/Douglas Kell
ChemBark
Realizations in Biostatistics
Chemjobber
Pharmalot
ChemSpider Blog
Pharmagossip
Med-Chemist
Organic Chem - Education & Industry
Pharma Strategy Blog
No Name No Slogan
Practical Fragments
SimBioSys
The Curious Wavefunction
Natural Product Man
Fragment Literature
Chemistry World Blog
Synthetic Nature
Chemistry Blog
Synthesizing Ideas
Business|Bytes|Genes|Molecules
Eye on FDA
Chemical Forums
Depth-First
Symyx Blog
Sceptical Chymist
Lamentations on Chemistry
Computational Organic Chemistry
Mining Drugs
Henry Rzepa


Science Blogs and News:
Bad Science
The Loom
Uncertain Principles
Fierce Biotech
Blogs for Industry
Omics! Omics!
Young Female Scientist
Notional Slurry
Nobel Intent
SciTech Daily
Science Blog
FuturePundit
Aetiology
Gene Expression (I)
Gene Expression (II)
Sciencebase
Pharyngula
Adventures in Ethics and Science
Transterrestrial Musings
Slashdot Science
Cosmic Variance
Biology News Net


Medical Blogs
DB's Medical Rants
Science-Based Medicine
GruntDoc
Respectful Insolence
Diabetes Mine


Economics and Business
Marginal Revolution
The Volokh Conspiracy
Knowledge Problem


Politics / Current Events
Virginia Postrel
Instapundit
Belmont Club
Mickey Kaus


Belles Lettres
Uncouth Reflections
Arts and Letters Daily
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline

In the Pipeline

« Getting van der Waals Forces Right | Main | Ariad in Limbo »

October 22, 2013

Size Doesn't Matter. Does Anything?

Email This Entry

Posted by Derek

There's a new paper in Nature Reviews Drug Discovery that tries to find out what factors about a company influence its research productivity. This is a worthy goal, but one that's absolutely mined with problems in gathering and interpreting the data. The biggest one is the high failure rate that afflicts everyone in the clinic: you could have a company that generates a lot of solid ideas, turns out good molecules, gets them into humans with alacrity, and still ends up looking like a failure because of mechanistic problems or unexpected toxicity. You can shorten those odds, for sure (or lengthen them!), but you can never really get away from that problem, or not yet.

The authors have a good data set to work from, though:

It is commonly thought that small companies have higher research and development (R&D) productivity compared with larger companies because they are less bureaucratic and more entrepreneurial. Indeed, some analysts have even proposed that large companies exit research altogether. The problem with this argument is that it has little empirical foundation. Several high-quality analyses comparing the track record of smaller biotechnology companies with established pharmaceutical companies have concluded that company size is not an indicator of success in terms of R&D productivity1, 2.

In the analysis presented here, we at The Boston Consulting Group examined 842 molecules over the past decade from 419 companies, and again found no correlation between company size and the likelihood of R&D success. But if size does not matter, what does?

Those 842 molecules cover the period 2002-2011, and of them, 205 made it to regulatory approval. (Side note: does this mean that the historical 90% failure rate no longer applies? Update: turns out that's the number of compounds that made it through Phase I, which sounds more like it). There were plenty of factors that seemed to have no discernable influence on success - company size, as mention, public versus private financing, most therapeutic area choices, market size for the proposed drug or indication, location in the US, Europe, or Asia, and so on. In all these cases, the size of the error bars leave one unable to reject the null hypothesis (variation due to chance alone).

What factors do look like more than chance? The far ends of the therapeutic area choice, for one (CNS versus infectious disease, and these two only). But all the other indicators are a bit fuzzier. Publications (and patents) per R&D dollar spent are a positive sign, as is the experience (time-in-office) of the R&D heads. A higher termination rate in preclinical and Phase I correlated with eventual success, although I wonder if that's also a partial proxy for desperation, companies with no other option but to push on and hope for the best (see below for more on this point). A bit weirdly, frequent mention of ROI and the phrase "decision making" actually correlated positively, too.

The authors interpret most or all of these as proxy measurements of "scientific acumen and good judgement", which is a bit problematic. It's very easy to fall into circular reasoning that way - you can tell that the companies that succeeded had good judgement, because their drugs succeeded, because of their good judgement. But I can see the point, which is what most of us already knew: that experience and intelligence are necessary in this business, but not quite sufficient. And they have some good points to make about something that would probably help:

A major obstacle that we see to achieving greater R&D productivity is the likelihood that many low-viability compounds are knowingly being progressed to advanced phases of development. We estimate that 90% of industry R&D expenditures now go into molecules that never reach the market. In this context, making the right decision on what to progress to late-stage clinical trials is paramount in driving productivity. Indeed, researchers from Pfizer recently published a powerful analysis showing that two-thirds of the company's Phase I assets that were progressed could have been predicted to be likely failures on the basis of available data3. We have seen similar data privately as part of our work with many other companies.

Why are so many such molecules being advanced across the industry? Here, a behavioural perspective could provide insight. There is a strong bias in most R&D organizations to engage in what we call 'progression-seeking' behaviour. Although it is common knowledge that most R&D projects will fail, when we talk to R&D teams in industry, most state that their asset is going to be one of the successes. Positive data tends to go unquestioned, whereas negative data is parsed, re-analysed, and, in many cases, explained away. Anecdotes of successful molecules saved from oblivion often feed this dynamic. Moreover, because it is uncertain which assets will fail, the temptation is to continue working on them. This reaction is not surprising when one considers that personal success for team members is often tied closely to project progression: it can affect job security, influence within the organization and the ability to pursue one's passion. In this organizational context, progression-seeking behaviour is entirely rational.

Indeed it is. The sunk-cost fallacy should also be added in there, the "We've come so far, we can't quit now" thinking that has (in retrospect) led so many people into the tar pit. But they're right, many places end up being built to check the boxes and make the targets, not necessarily to get drugs out the door. If your organization's incentives are misaligned, the result is similar to trying to drive a nail by hitting it from an angle instead of straight on: all that force, being used to mess things up.

Comments (30) + TrackBacks (0) | Category: Business and Markets | Drug Development | Drug Industry History


COMMENTS

1. Kelvin on October 22, 2013 7:46 AM writes...

Good clinical trials tell you exactly how and when to proceed, and when to quit. The only bad result is a fuzzy, uninformative one that builds false hope.

Permalink to Comment

2. Matthew Herper on October 22, 2013 8:19 AM writes...

On the failure rate: they probably didn't start with molecules on preclinical. If you're looking at molecules that got a company built around them or made it through a phase I trial or two, the failure rate should be cut. But this does mean it's not a totally representative sample, no?

Permalink to Comment

3. Anonymous on October 22, 2013 8:36 AM writes...

I wonder how many of these are me-too drugs that were approved, but failed in the market...

Permalink to Comment

4. SAR screener on October 22, 2013 8:50 AM writes...

I always thought that the 90% figure came from projects started not molecules?

Permalink to Comment

5. petros on October 22, 2013 9:22 AM writes...

it's 205 out of 842 that got through phase I if you look at the supplementary material

Permalink to Comment

6. Cellbio on October 22, 2013 9:26 AM writes...

Could higher termination rates in Phase I and preclinical also influence future success because there were several options but limited capacity to advance multiple programs and the "best" was chosen?

I would think a significant influence on choosing which program to bet on would be the available results from competitors' programs. If true, this would confound the data (along with me-toos) and indicate why innovation struggles as it is harder to advance unknowns than programs with external validation. It is certainly my experience that the industry wide pipeline was one of the major influencers of decision makers when evaluating which program to move forward.

Permalink to Comment

7. barry on October 22, 2013 9:33 AM writes...

"...many low-viability compounds are knowingly being progressed to advanced phases of development. "
Drug Discovery is built on research, and research is built on science and science is built on falsification. As in "this compound" (or this target) is not the way to treat that disease". If we're not designing, running and believing experiments that will kill weak candidates early, we'll go on spending $billions to see them killed later.

Permalink to Comment

8. Anonymous on October 22, 2013 9:33 AM writes...

"It is certainly my experience that the industry wide pipeline was one of the major influencers of decision makers when evaluating which program to move forward."

I wouldn't be surprised if that was the main decision driver: sheep mentality, follow vs lead and innovate

Permalink to Comment

9. Biotech Chemist on October 22, 2013 9:46 AM writes...

"team members is often tied closely to project progression: it can affect job security"

I see this as the biggest problem. It's pretty hard to kill your own project when your promotion and bonus are tied to it's success or failure. Whatsmore in our PC culture many leaders would rather let something fail in late stage rather than make a tough decision by cutting something early and then having a controversial decision on their hands.

Permalink to Comment

10. Anon on October 22, 2013 9:48 AM writes...

"There were plenty of factors that seemed to have no discernable influence on success - company size, as mention, public versus private financing, most therapeutic area choices, market size for the proposed drug or indication, location in the US, Europe, or Asia, and so on. In all these cases, the size of the error bars leave one unable to reject the null hypothesis (variation due to chance alone)"
This seems kind of predictable. Most VC, companies, investors etc. have been moving on these correlative unsubstantiated trends over the last 10 years. It makes sense that they finally hit an equilibrium.

Permalink to Comment

11. Anonymous on October 22, 2013 10:53 AM writes...

I'd like to see an analysis based on what percentage of research was off shored. Or maybe look to see if the average wage of the research staff influences success. Or maybe see what percentage of research staff were management vs. technical. I'd also like to see if the use of consulting groups like the one who published this study influenced the success rate.

Permalink to Comment

12. Pete on October 22, 2013 10:54 AM writes...

Somebody needs to tell the authors that correlation is not a yes/no quantity. Have a look at the figure captions.

Figure 1: Factors not correlated with success or failure in drug development

Figure 2: Factors correlated with success or failure in drug development

Permalink to Comment

13. Pete on October 22, 2013 11:01 AM writes...

I've linked a bit of light reading for the Boston Consulting Group as the URL for this comment (having botched it in the previous comment).

Permalink to Comment

14. Anonymous on October 22, 2013 11:42 AM writes...

"I'd also like to see if the use of consulting groups like the one who published this study influenced the success rate."

The answer is simple: Eroom's Law

So either they have not made any impact on declining R&D productivity, or quite possibly, they played a part in it!

Permalink to Comment

15. Andy on October 22, 2013 12:04 PM writes...

"We've come so far, we can't quit now"

Given exponential cost increases as development progresses, this makes no sense at all. To a first approximation I work with the assumption that next year's costs will equal the sum of all previous years' costs combined, so there isn't really much in the way of "sunk costs"; all of the previous spend is trivial compared to what's coming next.

Permalink to Comment

16. Cellbio on October 22, 2013 12:06 PM writes...

#14,

I was part of a major BCG initiative that totally destroyed R&D productivity. The result of their "work" was an elimination of all R folks from early development. Instead of domain experts and project legacy, they (along with company management) instituted a 5 person team with Clinical, Operations, Marketing, Manufacturing and Project Management. It was as if all one needed for success was to rid the room of uncertainty so that Gantt charts could be built with streamlined execution. The system fell apart in about a year. A total waste of millions and millions.

Permalink to Comment

17. Ann Onymous on October 22, 2013 12:36 PM writes...

"A higher termination rate in preclinical and Phase I"

Could be a sign of common sense and leadership, not just management. It takes guts to kill a project whereas letting it die of its own accord just takes an iron butt - sit still until it goes away.

Permalink to Comment

18. lynn on October 22, 2013 12:41 PM writes...

@barry #7 - I heartily agree. Falsify early! Do negative controls. Do the killer experiment! But this takes a reasonable amount of experience in the field, in screening, in med chem, toxicology - preclinically... in order to know what problems may arise, in order to design those killer experiments [as Cellbio #16 implies].

Permalink to Comment

19. CMCguy on October 22, 2013 2:06 PM writes...

Guess the overall data here does not support this but based on statement "frequent mention of ROI and the phrase "decision making" actually correlated positively" I would have expected there to be a real difference based on sizes of organizations. As far as ROI small companies can be wasteful and burn through their cash quickly enough however most are by comparison amateurs when it comes to the inherent big spending by bloated large pharmas just to maintain ongoing operations. Likewise in terms of any decision making smaller-flatter works better than exhaustive committees, managers, executive review cycles that is typically SOP as company grows.

Per #14 I too was speculating Consultants like BCG or worse McKinsey have a directly negative influence. Typical fancy Powerpoints with Quick and easy fixes offered for complex problems never seem to do more than generate a brief blip and then in longer term become distractions.

Permalink to Comment

20. Tim Terry on October 22, 2013 5:43 PM writes...

Part of the problem is that companies continue with failing compounds to keep their portfolio goals on tsrget. More realistically, cutting a compound when it begins to show difficulties would be good portfolio management, Allowing more money to available for new compounds and less spent on those attriting.

Unfortunately Big Pharm has a habit of letting senior directors pin their names to some compounds which pushes them past the point they ought to have been killed. This pushes up failure rate and continues to expend resources on projects that should not still be in the portfolio for development.

Permalink to Comment

21. Cellbio on October 22, 2013 6:41 PM writes...

Tim,

While I generally agree, one concept you mention can be problematic when practiced, namely, terminated compounds when they begin to show difficulties. Something always comes up when the tough tests start and deep scrutiny follows a compound through development. Switching to blue skies can mean rolling back to compounds whose challenges are yet unknown. It is a fine line between appropriate perseverance in the face of rising challenges and being willing to walk away early when there are good reasons to do so. Do this shuffle repeatedly and then see if the money people (management or VCs) have the appetite for more. In smallcos, VCs might let you change up once or twice if you are lucky. In bigcos, you might get to do this for a few years or more, but then major cuts comes as there is no ROI for cycling new compounds into Ph1 only to call it off. No easy answers I am afraid.

Permalink to Comment

22. Anonymous on October 22, 2013 6:56 PM writes...

@21: Cellbio - "there is no ROI for cycling new compounds into Ph1 only to call it off."

Not sure if that makes any sense.

Permalink to Comment

23. Gasoline on October 22, 2013 9:22 PM writes...

@11
I'd also like to see that!
"I'd like to see an analysis based on what percentage of research was off shored. Or maybe look to see if the average wage of the research staff influences success. Or maybe see what percentage of research staff were management vs. technical."

Permalink to Comment

24. Anonymous on October 22, 2013 9:35 PM writes...

I'd like to see the impact of Lean Six Sigma. Given that AZ have more Lean practitioners in R&D than any other pharma company and have barely had a single drug approved in years, I have a pretty good idea...

Also, to measure the impact of regular restructuring would be very interesting.

Permalink to Comment

25. Cellbio on October 22, 2013 10:13 PM writes...

What I meant to communicate is there is no value created for filling a pipeline with early candidates if this is the outcome of shutting down anything with warts. There is only value for taking risk and moving forward to definitive trials and approval.

This can be different for small companies, was quite so prior to 2008, but now seems to be tough to get value for Ph1 molecules.

Permalink to Comment

26. Slate on October 24, 2013 4:37 PM writes...

What is the point, let alone value, of this article?


Permalink to Comment

27. NRDDesist! on October 24, 2013 4:38 PM writes...

What is the point, let alone value, of this article?


Permalink to Comment

28. Anonymous on October 25, 2013 7:34 AM writes...

@26: To show that the consultants still can't appreciate that innovation and R&D productivity is driven by curiosity, courage and entrepreneurial spirit.

Permalink to Comment

29. PharmaGuy on October 26, 2013 6:49 AM writes...

Perhaps another factor influencing the advance of low-viability compounds in the R&D pipeline is financial; i.e., to improve stock performance in the short term. Financial reports focus on number of compounds that are in certain advanced stages of development. So there is incentive to move as many molecules forward as possible in order to provide a rosier picture to investors.

Permalink to Comment

30. Anonymous on November 13, 2013 4:37 AM writes...

In my point of view, when a compound is found to failing and is most likely not going to reach the market it’s better to cut it down or stop the R&D rather than waiting till the later stages. Although its takes lots of money it begin with a new compound, but wasting time, money and energy for a product which is going to fail isn’t worth it.

Permalink to Comment

POST A COMMENT




Remember Me?



EMAIL THIS ENTRY TO A FRIEND

Email this entry to:

Your email address:

Message (optional):




RELATED ENTRIES
Amicus Fights Its Way Through in Fabry's
Did Pfizer Cut Back Some of Its Best Compounds?
Don't Optimize Your Plasma Protein Binding
Fluorinated Fingerprinting
One of Those Days
ChemDraw Days
Incomprehensible Drug Prices? Think Again.
Proteins Grazing Against Proteins