Corante

About this Author
DBL%20Hendrix%20small.png College chemistry, 1983

Derek Lowe The 2002 Model

Dbl%20new%20portrait%20B%26W.png After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases. To contact Derek email him directly: derekb.lowe@gmail.com Twitter: Dereklowe

Chemistry and Drug Data: Drugbank
Emolecules
ChemSpider
Chempedia Lab
Synthetic Pages
Organic Chemistry Portal
PubChem
Not Voodoo
DailyMed
Druglib
Clinicaltrials.gov

Chemistry and Pharma Blogs:
Org Prep Daily
The Haystack
Kilomentor
A New Merck, Reviewed
Liberal Arts Chemistry
Electron Pusher
All Things Metathesis
C&E News Blogs
Chemiotics II
Chemical Space
Noel O'Blog
In Vivo Blog
Terra Sigilatta
BBSRC/Douglas Kell
ChemBark
Realizations in Biostatistics
Chemjobber
Pharmalot
ChemSpider Blog
Pharmagossip
Med-Chemist
Organic Chem - Education & Industry
Pharma Strategy Blog
No Name No Slogan
Practical Fragments
SimBioSys
The Curious Wavefunction
Natural Product Man
Fragment Literature
Chemistry World Blog
Synthetic Nature
Chemistry Blog
Synthesizing Ideas
Business|Bytes|Genes|Molecules
Eye on FDA
Chemical Forums
Depth-First
Symyx Blog
Sceptical Chymist
Lamentations on Chemistry
Computational Organic Chemistry
Mining Drugs
Henry Rzepa


Science Blogs and News:
Bad Science
The Loom
Uncertain Principles
Fierce Biotech
Blogs for Industry
Omics! Omics!
Young Female Scientist
Notional Slurry
Nobel Intent
SciTech Daily
Science Blog
FuturePundit
Aetiology
Gene Expression (I)
Gene Expression (II)
Sciencebase
Pharyngula
Adventures in Ethics and Science
Transterrestrial Musings
Slashdot Science
Cosmic Variance
Biology News Net


Medical Blogs
DB's Medical Rants
Science-Based Medicine
GruntDoc
Respectful Insolence
Diabetes Mine


Economics and Business
Marginal Revolution
The Volokh Conspiracy
Knowledge Problem


Politics / Current Events
Virginia Postrel
Instapundit
Belmont Club
Mickey Kaus


Belles Lettres
Uncouth Reflections
Arts and Letters Daily
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline

In the Pipeline

« Cambridge "In the Pipeline" Reader Meetup: October 10 | Main | Did the Biotech IPO Window Just Close? »

October 8, 2013

Forecasting Drug Sales: Har, Har.

Email This Entry

Posted by Derek

You're running a drug company, and you have a new product coming out. How much of it do you expect to sell? That sounds like a simple question to answer, but it's anything but, as a new paper in Nature Reviews Drug Discovery (from people at McKinsey, no less) makes painfully clear.

Given the importance of forecasting, we set out to investigate three questions. First, how good have drug forecasts been historically? And, more specifically, how good have estimates from sell-side analysts been at predicting the future? Second, what type of error has typically been implicated in the misses? Third, is there any type of drug that has been historically more easy or difficult to forecast?

The answer to the first question is "Not very good at all". They looked at drug launches from 2002-2011, a period which furnished hundreds of sales forecasts to work from. Over 60% of the consensus forecasts were wrong by 40% or more. Stop and think about that for a minute - and if you're in the industry, stop and think about the times you've seen these predictions made inside your own company. Remember how polished the PowerPoint slides were? How high-up the person was presenting them? How confident their voice was as they showed the numbers? All for nothing. If these figures had honest error bars on them, they'd stretch up and down the height of any useful chart. I'm reminded of what Fred Schwed had to say in Where Are the Customers' Yachts about stock market forecasts: "Concerning these predictions, we are about to ask: 1. Are they pretty good? 2. Are they slightly good? 3. Are they any damn good at all? 4. How do they compare with tomorrow's weather prediction you read in the paper? 5. How do they compare with the tipster horse race services?".
Forecasts.png
As you can see from the figure, the distribution of errors is quite funny-looking. If you start from the left-hand lowball side, you think you're going to be looking at a rough Gaussian curve, but then wham - it drops off, until you get to the wildly overoptimistic bin, which shows you that there's a terribly long tail stretching into the we're-gonna-be-rich category. This chart says a lot about human psychology and our approach to risk, and nothing it says is very complimentary. In case you're wondering, CNS and cardiovascular drugs tended to be overestimated compared to the average, and oncology drugs tended to be underestimated. That latter group is likely due to an underestimation of the possibility of new indications being approved.

Now, those numbers are all derived from forecasts in the year before the drugs launched. But surely things get better once the products got out into the market? Well, there was a trend for lower errors, certainly, but the forecasts were still (for example) off by 40% five years after the launch. The authors also say that forecasts for later drugs in a particular class were no more accurate than the ones for the first-in-class compounds. All of this really, really makes a person want to ask if all that time and effort that goes into this process is doing anyone any good at all.

Writing at Forbes, David Shaywitz (who also draws some lessons here from Taleb's Antifragile) doesn't seem to think that it is, but he doesn't think that anyone is going to want to hear about it:

Unfortunately, the new McKinsey report is unlikely to matter very much. Company forecasters will say their own data are better, and will point to examples of forecasts that happen to get it right. They will emphasize the elaborate methodologies they use, and the powerful algorithms they employ (all real examples from my time in the industry). Consultants, too, will continue to insist they can do it better.

And indeed, one of the first comments that showed up to his piece was from someone who appears to be doing just that. In fact, rather than show any shame about these numbers, plenty of people will see them as a marketing opportunity. But why should anyone believe the pitch? I think that this conclusion from the NRDD paper is a lot closer to reality:

Beware the wisdom of the crowd. The 'consensus' consists of well-compensated, focused professionals who have many years of experience, and we have shown that the consensus is often wrong. There should be no comfort in having one's own forecast being close to the consensus, particularly when millions or billions of dollars are on the line in an investment decision or acquisition situation.

The folks at Popular Science should take note of this. McKinsey Consulting has apparently joined the "War on Expertise"!

Comments (32) + TrackBacks (0) | Category: Business and Markets | Drug Development


COMMENTS

1. oldnuke on October 8, 2013 8:15 AM writes...

If the forecasts are that wildly inaccurate, how in the world do they plan their manufacturing?

Permalink to Comment

2. Anonymous on October 8, 2013 8:19 AM writes...

@1: Learn very fast during product launch!

Permalink to Comment

3. newnickname on October 8, 2013 8:24 AM writes...

Frequently, the forecasting begins even before there's a product to launch! Sometimes even before the first synthetic step! ("If we can make an inhibitor of X to cure Y, it will be a billion dollar molecule!")

Permalink to Comment

4. debunk on October 8, 2013 8:27 AM writes...

@ 1...Do not worry! The big companies will ignore this paper and it will be business as usual.

Permalink to Comment

5. simpl on October 8, 2013 8:54 AM writes...

There is an additional complexity, which is that internal forecasts vary wildly (factor 100-300%) before launch. We run an optimistic and a pessimistic forecast for that reason.
The consequences of being wrong are much more serious if too low than too high - a year-long stock-out kills sales, but a year's extra inventory is a financial inefficiency. That is probably the main cause of the high peak, as experienced launch managers don't want to get caught short of supply.
Also, don't forget each launch varies massively - we have inhaled drugs or antibodies which are small volume and valuable, and simple cardiovascular combinations where the amounts are huge, cheap, and relatively predictable.

Permalink to Comment

6. Carlos at Lacerta Bio on October 8, 2013 9:01 AM writes...

The study uses sell-side analysts' estimates, not internal pharma forecasts. So it's a stretch to say that internal pharma forecasts are this bad, especially once a product is launched. There is no doubt that a forecast for a product that's 10 years away will have a lot of variability in hindsight. But internal pharma/biotech forecasts for near-launch and on-market products are not this far off.

If anything, I'd question the wisdom of the investors who use sell-side analyst forecasts to calculate company valuations and stock prices.

Permalink to Comment

7. CMCguy on October 8, 2013 9:01 AM writes...

#1 Manufacturing Planners hold their breaths (or perhaps noses) when they see Marketing Estimates but typically gear capacity toward the max as is easier to slow down than speed up. In most cases production has certain gating factors, typically associated with expenditures, that are independent of Marketing and one hopes they provide what is necessary for success.

Permalink to Comment

8. petros on October 8, 2013 9:37 AM writes...

not really surprising since there are so many variables that influence the real outcomes. It would have been helpful to see some real examples.

I'd bet that no one forecast Glivec to be a major blockbuster and I'd be very surprised if any analysts came close to predicting how big Lipitor would be before it had hit the market.

Then there are the turkeys, which are hyped up as superblockbusters and for various reasons are commercial flops.

Permalink to Comment

9. Anonymous on October 8, 2013 9:40 AM writes...

I disagree with the last quote. This is not the wisdom of the crowd nor the consensus. And it wasn't made by focused professionals with many years of experience. I think that chart shows you what happens when you let a bunch of twits with marketing degrees who struggled to get through algebra run your company, choose your therapeutic areas and predict your sales.

If you want to predict what drug sales will be, sit down with the patients and find out what they deal with every day. Then you might get some better accuracy in those predictions. Don't listen to some 25 year old who has never been sick a day in his life.

Permalink to Comment

10. annonie on October 8, 2013 9:51 AM writes...

New drugs for treatments that have no drugs---little market background to base acceptance, sales, etc. Just go with it & hope you've got a winner!

Permalink to Comment

11. Kelvin on October 8, 2013 9:51 AM writes...

The key is not to get more accurate forecasts, because their accuracy will always be limited by the lack of available information, but to quantify the degree of uncertainty (and thus risk) in the forecasts, in order to make better decisions. This is no different from giving an error bar to qualify the probability distribution of any other assumption, expectation or result.

Permalink to Comment

12. barry on October 8, 2013 9:56 AM writes...

I am reminded of the pre-launch forecast of cimetidine/Tagamet. The BusinessDevelopment people estimated what fraction of the (existing) Maalox market they could grab and came out with a number near $100million/annum. That was off by 1,000%. In retrospect, it's clear that any disorder for which there is no effective treatment will be under-reported. But ulcer patients came out of the woodwork as soon as the word was out that this stuff works.
This sort of error had real consequences. Smith-Kline cut off chemistry work on expanding the patent protection. Very shortly, although Cimetidine was earning over $1billion/annum (the industry's first "blockbuster") Ranitidine (which is structurally very very similar and clinically indistinguishable) was earning $2.3billion/annum.
And it's relevant today because BigPharma's mega-corporations feel they can only afford to pursue blockbusters. So they rely on just this sort of pre-release Bus. Dev. bullshit and they kill projects that might indeed have turned out to be big winners.
The shining counter-example is Gleevec. That was brought to market as an orphan drug. I.e. everyone acknowledged that it had a tiny market potential. But it quickly became a blockbuster. And Novartis (or was it still Ciba?) developed it only under duress from Richard Pazdur of the FDA.

Permalink to Comment

13. Calvin on October 8, 2013 9:59 AM writes...

#8 petros

John LaMattina has described on his blog how Liptor was predicted to maybe make $600M in peak sales from the internal group at Pfizer. He has used that experience to point out how ineffective such predictions are. Nice in that case though to get it so wrong!

#6 It's a nice thought but there are too many examples of internal forecasts from within pharma being equally inaccurate. Exubera, Brillinta, Provenge. It's still finger in the air stuff.

Permalink to Comment

14. Electrochemist on October 8, 2013 10:01 AM writes...

#9 is spot on. Sales projections for a new molecule start during early development, and take off from there. Funding for development is tied (at least partially) to projected sales (net present value at launch and then 5 years post-launch). Thus, there is no motivation for those making the projections to be conservative (or even accurate), and plenty of motivation to grossly exaggerate. (Who wouldn't want their own team's project to be funded?) The data in the bar graph validate this.

Permalink to Comment

15. petros on October 8, 2013 10:03 AM writes...

Calvin

I hadn't seen that comment but knew of external forecasts that weren't that high. Interesting that Pfizer was so pessimistic.

Permalink to Comment

16. Anonymous on October 8, 2013 10:04 AM writes...

@5: Good point about needing to oversupply to avoid stock-outs during launch, but oversupplying on good forecasts is not the same as making bad forecasts!

Permalink to Comment

17. sigma147 on October 8, 2013 10:10 AM writes...

Not in pharma, but in Dx field - we're usually expecting marketing's projections to be off by about an order of magnitude (usually low, but sometimes high) for just about everything they project. Heck, even the customers don't necessarily know what they want. Just refer to the "Innovator's Dilemna" for a discussion of how marketing predictions and disruptive technology work (and it isn't exactly hand-in-hand). The point is that many factors contribute to market demand, and quite a few of those are outside the realm of controllable (or even knowable). Care to predict how the ACA will affect market demand for your newest cardio drug? Yeah, me neither...

Permalink to Comment

18. flem on October 8, 2013 11:20 AM writes...

a forecast is only as good as the assumptions it uses. for new drugs entering an existing class (eg. statins) you typically apply a penetration model based on historical observations in an analgous class. The really hard part is applying assumptions to a completely new class. Imagine the wild estimates and assumptions that are needed to determine sales for a new alzheimers drug? particularly if needs to be used for primary prevention. Nevermind what the forecasts where for Lipitor, I remember the forecast for Mevacor... they were hard to believe at the time given all the unknowns..but turned out pretty on the mark. Of course when 4S came into play the forecasts took on a very different look.

Permalink to Comment

19. Em on October 8, 2013 11:34 AM writes...

I used to work on automation projects for medical devices, and the most honest thing I've ever heard from a customer when discussing expected manufacturing volumes was "These are the marketing estimates for annual sales. The only thing we know about them is that they're wrong."

Permalink to Comment

20. Lyle Langley on October 8, 2013 11:58 AM writes...

@#9,
"If you want to predict what drug sales will be, sit down with the patients and find out what they deal with every day. Then you might get some better accuracy in those predictions. Don't listen to some 25 year old who has never been sick a day in his life."

I would HIGHLY doubt this approach would give any type of accuracy in predicting financial success. This would the be the reason a group/company goes after a target/disease due to unmet medical need. Unless all of the patients you talk to pay out-of-pocket, they don't determine whether or not a drug has robust sales. The payer is who needs to be questioned, for if it's not reimbursed, the sales won't be there.

Permalink to Comment

21. alig on October 8, 2013 12:26 PM writes...

It's been awhile since I saw the Glaxo slide predicting Advair sales, but it was something like 150 mil/yr on the low, 300 mil/yr expected and 800 mil/yr was the pie-in-the-sky peak estimate. It still sells 4-5 billion/yr. On the other side they forecasted Levitra to sell 2 billion/ yr when they inlicensed it from Bayer. I don't think it ever sold 500 mil/yr. Internal estimates perform as poorly as the consensus estimates.

Permalink to Comment

22. AS on October 8, 2013 12:52 PM writes...

Most of the comments seem to be missing the point - these are Wall Street sell-side analyst forecasts, not the companies themselves. Having been an equity research analyst myself in a former life and now in the pharma industry, I can tell you that the majority of the analysts do not know what they're doing when it comes to forecasting. They don't have the budget to understand the marketplace (not the stock market), nor do they have the commercial experience to understand the subtleties of selling a drug. They are not paid to be an accurate forecaster - they are paid to make the right stock call, and the forecast needs to be directionally right, though it may not matter in the long run. Then I could go on and on about the pressure to be positive on companies that are current or potential banking clients...

Permalink to Comment

23. AS on October 8, 2013 12:53 PM writes...

Most of the comments seem to be missing the point - these are Wall Street sell-side analyst forecasts, not from the companies themselves. Having been an equity research analyst myself in a former life and now in the pharma industry, I can tell you that the majority of the analysts do not know what they're doing when it comes to forecasting. They don't have the budget to understand the marketplace (not the stock market), nor do they have the commercial experience to understand the subtleties of selling a drug. They are not paid to be an accurate forecaster - they are paid to make the right stock call, and the forecast needs to be directionally right, though it may not matter in the long run. Then I could go on and on about the pressure to be positive on companies that are current or potential banking clients...

Permalink to Comment

24. gwern on October 8, 2013 1:56 PM writes...

Copied: https://dl.dropboxusercontent.com/u/182368464/2013-cha.pdf

Makes me wonder: if the sell-side analysts are so bad, what are the internal projections in the hedge funds like?

Permalink to Comment

25. SE on October 8, 2013 2:55 PM writes...

This is burning a Straw Man.

Rule #1 for Sell-Side Analysts: do NOT write a "Sell" recommendation. You want clients to buy the stock. That way, your market is those clients who do NOT own the stock; a much bigger market than those who already own it (this is textbook stuff btw). Therefore, accentuate the positive.

Besides, new regulations nothwithstanding, Rule #2 is do not write reports that piss off companies that might do business with the investment banking side of the firm.

Permalink to Comment

26. Anonymous on October 8, 2013 4:07 PM writes...

Rule #3 is do NOT talk about fight club!

Permalink to Comment

27. David Shaywitz on October 8, 2013 7:59 PM writes...

Re: Comment 25 by SE: your critique is that sell side analysts are biased, and systematically less critical than they should be. While this may be true, it's not what the McK data suggest; their point isn't that estimates are consistently inflated, but rather they are consistently inaccurate, and ~randomly sprayed, winding up far above the actual sales some times! way below at other times.

In any case, what intrigues me most is not specific issue of whether sell-side launch estimates accurate, but rather extent to which forecasting can and should reasonably be used to inform portfolio decisions - especially early on.

While I believe it can be instructive to go through exercise to be sure opportunity has received systematic and comprehensive considerations, I suspect more often than not, such forecasts are in a domain Taleb would say generates sterile information, which by virtue of false precision can be surprisingly dangerous and destructive to an organization; see: http://financialservices.house.gov/uploadedfiles/071411nassim.pdf .

Permalink to Comment

28. boo on October 9, 2013 12:37 AM writes...

Coincidently, I'm at a conference this week where a bunch of healthcare experts are bandying about some awfully rosy financial projections for the current healthcare reforms.

Permalink to Comment

29. Anon on October 9, 2013 7:40 AM writes...

An interesting Catch-22 when you think about it.

This consulting firm has done a thorough investigation.

And based on our analysis you shouldn't put any credence on anything we tell you.

But you should listen to us this time, because this time it's different.

Permalink to Comment

30. MikeC on October 9, 2013 10:46 AM writes...

I was quite surprised by figure 2: for large pharmas the variance remained large but the bias across 89 projections was only +1%. That means an equal (weighted) number of pessimists as optimists. Are there a lot of sell-side analysts who, contrary to SE #25, cater to potential clients looking to short big pharmas as opposed to investing in them? The variance of 64% for big pharma is also better than I expected: they're all at least in the ballpark for the size of the market, even if they're rolling dice on penetration.

Regardless, I love that this hit the news just after Merck announced it will be "improving" its business model by having its marketing prediction wonks climb even further up their R&D staffs',er, noses.

Permalink to Comment

31. mean spirit on October 9, 2013 11:54 AM writes...

I hadn't concerned myself about the investment consultant conflicts, but it did cross my mind that the article comes close to being an apology for the mess after the consultative planners had done the rounds on inventory minimisation programmes (2005-10?). The basic premise was that internal forecasts carry hidden bias, so listen to consultants, ignore and resist internals. It took us 18 months with extra shifts to catch up after that.

Permalink to Comment

32. Nirmal on October 10, 2013 2:45 AM writes...

If I were to make the chart showing showing distribution of errors, I'd do it on a log scale.

Intuitively to me, "less than -80%" (lower than 1/5th) should be compared to "more than +500%" (more than 5 times). -80% to -60% should be compared to +250% to +500%, etc. Then the chart might not look so odd and while the "tail" might still be long, it may not be "terribly long".

Of course, this does not change the reliability (or not) of these forecasts.

Permalink to Comment

POST A COMMENT




Remember Me?



EMAIL THIS ENTRY TO A FRIEND

Email this entry to:

Your email address:

Message (optional):




RELATED ENTRIES
XKCD on Protein Folding
The 2014 Chemistry Nobel: Beating the Diffraction Limit
German Pharma, Or What's Left of It
Sunesis Fails with Vosaroxin
A New Way to Estimate a Compound's Chances?
Meinwald Honored
Molecular Biology Turns Into Chemistry
Speaking at Northeastern