About this Author
DBL%20Hendrix%20small.png College chemistry, 1983

Derek Lowe The 2002 Model

Dbl%20new%20portrait%20B%26W.png After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases. To contact Derek email him directly: Twitter: Dereklowe

Chemistry and Drug Data: Drugbank
Chempedia Lab
Synthetic Pages
Organic Chemistry Portal
Not Voodoo

Chemistry and Pharma Blogs:
Org Prep Daily
The Haystack
A New Merck, Reviewed
Liberal Arts Chemistry
Electron Pusher
All Things Metathesis
C&E News Blogs
Chemiotics II
Chemical Space
Noel O'Blog
In Vivo Blog
Terra Sigilatta
BBSRC/Douglas Kell
Realizations in Biostatistics
ChemSpider Blog
Organic Chem - Education & Industry
Pharma Strategy Blog
No Name No Slogan
Practical Fragments
The Curious Wavefunction
Natural Product Man
Fragment Literature
Chemistry World Blog
Synthetic Nature
Chemistry Blog
Synthesizing Ideas
Eye on FDA
Chemical Forums
Symyx Blog
Sceptical Chymist
Lamentations on Chemistry
Computational Organic Chemistry
Mining Drugs
Henry Rzepa

Science Blogs and News:
Bad Science
The Loom
Uncertain Principles
Fierce Biotech
Blogs for Industry
Omics! Omics!
Young Female Scientist
Notional Slurry
Nobel Intent
SciTech Daily
Science Blog
Gene Expression (I)
Gene Expression (II)
Adventures in Ethics and Science
Transterrestrial Musings
Slashdot Science
Cosmic Variance
Biology News Net

Medical Blogs
DB's Medical Rants
Science-Based Medicine
Respectful Insolence
Diabetes Mine

Economics and Business
Marginal Revolution
The Volokh Conspiracy
Knowledge Problem

Politics / Current Events
Virginia Postrel
Belmont Club
Mickey Kaus

Belles Lettres
Uncouth Reflections
Arts and Letters Daily
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline

In the Pipeline

« Fraud: How, and Why, and How Again | Main | What Results Did You Have In Mind? »

March 20, 2009

Drug Industry Research: Reliable or Not?

Email This Entry

Posted by Derek

So, in light of the Reuben scandal of forged data about pain management in surgery patients, the question naturally comes to mind: how much role did industry play? I’ve seen articles (and had comments here) to the effect that industry-sponsored research is worthless: discount it, can't trust it, bought and paid for, and so on.

The problem is, you can't completely shake that accusation. Industries (and not just the drug industry, by any means) are willing to pay for results that tell them what they want to hear. And while at times that's crossed over into outright fraud, many times it's just that you can set up all kinds of studies, in all kinds of ways, and get all kinds of answers. Run enough of them, and you can choose the ones you like and pretend the others aren't there.

The whole idea of scientific research is that you don't operate like this, of course, and eventually these things do get settled out. If the drug industry really did make sure that only happy results came out, we'd never have catastrophic clinical trial failures, and never have any drugs recalled from the market. And things like the (Nobel-worthy) H. pylori story behind stomach ulcer formation never would have seen the light of day if the industry were capable (on the other hand) of burying everything it didn't want to hear about.

But there are biases, real and potential, and they always have to be looked out for. One error, though, is to assume that these biases can be eliminated by turning to academic research instead. That's the point of a recent Op-Ed in the Washington Post by David Shaywitz, who's worked both sides of the business:

Part of the problem is that we've been conditioned to trust university research. It is based, after all, on the presumably lofty motives of its practitioners. What's not to like about science carried out by academics who have nobly dedicated their lives to understanding the unknown, furthering knowledge and serving humanity?

. . .University researchers are in a constant battle for recognition and the rewards associated with success: research space, speaking engagements, funding and autonomy. Consequently, while academic research is often described as "curiosity-driven," the reality is messier, as (curiously) many researchers tend to pursue the trendiest technologies and explore topics that happen to be associated with the most generous levels of research support.

Moreover, since academic success is determined almost exclusively by the number and prestige of research publications, the incentives to generate results are exceedingly powerful and can encourage investigators to see patterns that may not exist, to disregard contradictory observations that might be important, to overvalue data that might be preliminary or unreliable, and to embrace conclusions that deserve to be viewed with far greater skepticism.

Shaywitz goes on to make the same point I did above - that the system is ultimately self-correcting - but is calling for people to recognize that academic research is also done by human beings, with all that entails. John Tierney at the New York Times had taken up this topic last fall, and wondered about what would happen if enough researchers decided to stop taking industry funding because they were tired of having their integrity questioned.

Tierney's responded to the Shaywitz piece now as well. The comments from his readers are all over the place each time. Some of them are (correctly, to my mind) going along with the idea that research always comes in with various potential biases and agendas, and should be judged case-by-case no matter the source. There are, naturally, some who aren't buying anything that might get industrial research off the hook.

"In industry sponsored comparative studies of medical treatments, the sponsor’s product always comes out on top," says one commenter there. But that's not true. I can give you plenty of examples right off the top of my head. For sure, we try to run studies that will show a benefit for our therapies - but we also have to pin these down to the real world for people (and the FDA) to have a better chance of trusting the results. We're not going to set up a trial that we have good reason to think will fail: life is too short, and the supply of funds is not infinite. You target the diseases (and the patients) that you think will benefit the most (and show the most impressive results, naturally).

And that's a bias to consider right there: we don't set up our trials randomly, so keep that in mind. But no one sets up drug trials randomly, anywhere. There's always a reason to do something so expensive and time-consuming - you should always keep that in mind, weigh it in your calculations, and decide from there.

Comments (16) + TrackBacks (0) | Category: Clinical Trials | Press Coverage | The Dark Side | Why Everyone Loves Us


1. Hap on March 20, 2009 12:10 PM writes...

1) People are willing to do lots of things for nonself-interested reasons (because they belive it to be true, for example, or because it's right - see Andrew Wakefield, I think), and to gain undeserved status is also a reason people commit fraud, but money is a more general reason for people to commit fraud, because almost everyone responds to it positively. To imply that the powers of financial and nonfinancial causes to motivate fraud are equivalent in theory or practice may not be accurate.

2) The self-correction depends on the quality of the original data and on its replication. If people don't know contrary data exists (because it isn't published because it's contrary), people are probably less likely to attempt to replicate. If studies are facilitated by industrial money or access, others would be less likely to be able to reproduce them. In some cases, the studies are big enough that it would be difficult and expensive for others to attempt to reproduce them (I suggested elsewhere that large total syntheses - norzoanthamine for example - might fit in that category as well because of the expense and time needed to reproduce them and the unlikeliness of funding to do so). If no one can try to reproduce the studies, for one reason or another, then the self-correction ability of science (at least in those particular cases) is a mirage, and the knowledge based on those studies is more contingent than it appears.

3) It seems that the funding companies might do better to watch those whose research they pay for, since the arrow of legitimacy goes both ways (the researchers are also trying to gain legitimacy by pharma funding). People such as Nemeroff (who didn't commit fraud but simply lied about/elided the amounts of his pharma compensation) damage the reputations of pharma companies, and it doesn't seem that whatever help they provided is worth the negative repercussions when they are dishonest or fradulent. Inhibiting that with better oversight might help.

4) The overselling of drugs and the lack of productivity of pharma companies probably contributes to the impact of fradulent doctors on pharma companies - people's mistrust of pharma leads them to look at events in a negative light, and the lack of good things that they seem from them means that no good is obvious to mitigate those impressions. If pharma was better trusted, the impact of people such as this would be far less.

Permalink to Comment

2. CMC Guy on March 20, 2009 12:29 PM writes...

As suggested in response to previous post IMO Academic research may be even more susceptible to "mistakes" because most Profs operate as "lone-wolf" researcher which has less of the direct checks and balances than in normal Industry Research (in-house or sponsored). However what it comes down to is that most R&D can be messy because of the complexities of the questions, experiments and especially the human factors. There is a false perception that Science can solve every problem and while I advocate is usually provides a reasonable approach(s) it is rarely smooth and straightforward process (particularly for new medicines). Unfortunately the stakes are so high when dealing with real people/real diseases.

I do think most errors are not from overt acts (as Rueben's were) but rather as is extremely well stated "the incentives to generate results are exceedingly powerful and can encourage investigators to see patterns that may not exist, to disregard contradictory observations that might be important, to overvalue data that might be preliminary or unreliable, and to embrace conclusions that deserve to be viewed with far greater skepticism". This applies to Industry researcher strongly also albeit with different motivation from publications. I would guess most Scientists have experienced some "self delusion" and is typical it takes your Prof/Supervisor, co-workers and even consultants that make you rethink positions/data.

Permalink to Comment

3. Mat Todd on March 20, 2009 7:47 PM writes...

Can't argue too much with Shaywitz's original piece. It also ends well: "Only by discovering our inner scientist can we fully delight in the hope of new research without being seduced by its charms."

I also like the quote I hadn't heard before: "university politics are so vicious precisely because the stakes are so small" (attributed to Henry Kissinger, but possibly incorrectly

Permalink to Comment

4. zts on March 21, 2009 8:52 AM writes...

This reminds me of an interview I heard on the radio a while ago. I forget the name, but the interviewee was an advocate for vaccines, and was speaking out against the people who believe vaccines cause autism and other problems.

A woman called in and said that if he could provide one study, just one study, that showed that these vaccines were safe, then she would believe him. He responded that he could name *dozens* of such studies off the top of his head. But she completely dismissed them, saying she could never trust any industry funded studies and demanded "independent" studies. He then mentioned something about the government, and she said that should would never trust anything the government said, and that she demanded truly "independent" studies.

He dealt with her pretty well and established that people could basically trust these studies, that they had enough checks and balances and whatnot, and that no third party groups were going to spend hundreds of millions of dollars to repeat studies that were competently and trust-worthily designed and executed the first time around. Suffice it to say, there was no amount of evidence that would cause her to rethink her ideas. I admit this was a more extreme example, but I think there are a lot of people out there who have fundamental problems trusting anything funded by big pharma, lumping us into the "evil" category with the likes of big tobacco and big oil. I think this is a major problem, and I don't really know how to address it, but I think it is vitally important for our industry that we do address this and improve our image.

Permalink to Comment

5. Sili on March 21, 2009 9:03 AM writes...

Wakefield is a rather poor example of selflessness given how he has been paid by antivax advocates to the tune of £500k right from the start.

I'll try not to blame big-I Industry, but it seems to me that there is an incentive for burying the worst results and generally massaging the data to put one's own product in the best possible light. Worst of all, as Ben Goldacre has repeatedly highlighted when discussing the research of others, only a fraction of all approved human trials are ever made public. And of course there's the publication bias - even if one wanted to get a negative result out, it's hard to get it into one of the 'good' journals.

I agree with BG that there need to be more transparency in all human trials. That includes making the raw data public within a given time after the end of the trial.

Do you think there's any way for Industry to dissociate itself from this poor image? Would it be possible to 'outsource' trials to some form of independent third party? I guess it would have to be paid for its services up front to minimize the room for economic pressure. But there'd presumably still be the threat of going elsewhere the next time, if a given trial doesn't give the desired result. Would it make sense to let the FDA run the trials from start to finish, rather than 'just' evaluate the results submitted to them?

And ... and I guess this is blaiming Industry ... is there any interest in getting rid of the poor image?

Permalink to Comment

6. Hap on March 22, 2009 3:34 PM writes...

I wouldn't put most of the noneconomically motivated people (of which I thought Wakefield was an example) in the "selfless" category - they want to impose themselves and their beliefs on lots of people. It's just that I thought he wasn't being motivated by cash. Silly me.

Where do the antivax people get their cash? I wonder if people suing vaccine companies might have something to do with them.

Permalink to Comment

7. Malcolm on March 22, 2009 6:46 PM writes...

I don't believe there's a magic bullet that will let good science triumph over bad.

Research that is funded and controlled by vested interests will be subject to bias. Leaving aside frank dishonesty, we still have publication bias and perhaps most importantly of all, a biased research agenda and biased research questions. Only rarely do companies commission research without good odds of a (commercially) positive outcome.

The International Committee of Medical Journal Editors has been doing great work combating the worst forms of publication bias, through promoting registration of trial, protocol and even results. Of course, this isn't enough.

There needs to be a critical mass of good quality, adequately funded non-industry research or we haven't a chance. And of course that research needs to be subject to the highest standards of transparency, peer scrutiny and public debate. Individual researchers may have their own agendas, but a diverse and dynamic public research community backed by pro bono research institutions can act as a corrective.

Government-funded research comes with biases to be sure, but has the advantage of working via systems and institutions that reinforce independence and scientific rigour, and that are largely open to public scrutiny.

As they say, the price of freedom is eternal vigilance.

Permalink to Comment

8. CMC guy on March 22, 2009 11:15 PM writes...

Malcolm why is bias for "vested interest" by industry any "worse" from the biases from academic, foundations, and government sources. Yes industry is often more directed and focused on outcomes (are you willing to invest in something you think will fail?). Sure it may not have some of the same open access because of legitimate commercial interests but that doesn't mean it lacks independence and scientific rigor. I think we need healthy research actions and interactions of all groups (Gov, Univ, Foundations, and Industry[large and small players]) to address the problems in drug R&D. Some people wish to eliminate or hamstring Industry because of bad events without recognizing the contributions.

Permalink to Comment

9. Malcolm on March 22, 2009 11:37 PM writes...

Without industry-conducted registration trials we would not have safe and effective drugs. I agree that these studies are done honestly and efficiently, and it would be foolish to "eliminate" it (though I'm happy that the way it's done is "hamstrung" by regulators to a large extent).

I am arguing that industry sponsored research in the absence of a healthy non-industry research base leads to bad medicine, or shall we say sub-optimal medicine.

Industry bias tends to be all in the same direction, for one thing (e.g. in favour of drug treatment). Given that head-to-head studies are uncommon, there is not much chance that competition between companies can have the salutary effect it might in other areas.

Industry-funded research is only subject to as much public scrutiny as regulation requires. Compare this to research in academia where proposals are considered by grant committees, and protocols by referees, to give two examples. To risk an analogy, industry funded research is a command economy and publicly funded science is noisy democracy...

Permalink to Comment

10. Jonadab the Unsightly One on March 23, 2009 8:13 AM writes...

Honestly, how much stock do you put in ANY research that hasn't been independently verified at least a couple of times? Nobody with a modicum of discernment is going to draw a whole lot of conclusions from just one study, other than along the lines of, "Well, that's interesting; someone should probably look into that further."

Yeah, studies sponsored by a company that has something to gain from a certain outcome are suspect; that's true in any industry, not just pharmaceuticals. But if the study is bogus in some way, aren't you generally going to find that out when another group tries to independently reproduce? Or, going the other direction, if the results are real, shouldn't they be reproducible by third parties?

Permalink to Comment

11. Malcolm on March 23, 2009 4:06 PM writes...


Most clinical studies performed by pharmaceutical companies are never independently reproduced.

An added difficulty is that we often want to pool all the research together to generate a 'super-sample' i.e. in a meta-analysis. This allows you to make a more precise estimate of the true effect size, or incidence of rare adverse events.

When you throw together a bunch of biased studies (or a biased sample of studies, if that's not too mind-bending) your pooled estimates look more precise but in fact still contain all the bias. Bad if you want to make rational decisions about which treatment to choose.

Permalink to Comment

12. CMC Guy on March 23, 2009 10:07 PM writes...

Malcolm I still see an unjustified bias against Industry research vs other forms that IMO is more perception than reality.

Per following points:
"Industry-funded research is only subject to as much public scrutiny as regulation requires." Yes there is often lack of and delays in "publishing" but that is nature of competative business. Also if used to support a drug application it is reviewed by Regulators (Who are aware of the study bias and sometimes create it by forcing certain studies/statistical measures for approval)

"Compare this to research in academia where proposals are considered by grant committees, and protocols by referees, to give two examples." Please direct me to where this info is public as grants and protocol reviews are confidential. The reviewers are typically other academics who have own agenda's or areas to promote. In most cases industry studies are reviewed by wider scopes of disciplines and yes hired consultants.

"To risk an analogy, industry funded research is a command economy and publicly funded science is noisy democracy." Again Industry research is directed toward specific aims and although less democratic (especially at some companies) that does not lessen the legitimate nature of the work. Sorry but public funding does not automatically make it better (especially in "old boys network" that controls where monies directed) and there is much noise from "pork-barrel" funding with questionable value (mating habits of ear wicks).

I think we do agree that we need all kinds of contributions from all sectors but those who automatically write off value of Industry because is tainted by "profits" seem to ignore how funding in academia and government occurs.

Permalink to Comment

13. Malcolm on March 23, 2009 11:20 PM writes...

@CMC guy

Academic research is of course imperfect, and is perhaps less efficient than research performed under contract, for example. And I'm all for increasing the transparency, rigour and accountability of academic research funding.

I'm not painting a black and white picture here. Perhaps the most important point I want to make is that commissioning bias exists, is important, and is a significant risk with industrial research. The studies themselves may be conducted with utmost honesty, but this is not enough to eliminate the problem.

Permalink to Comment

14. srp on March 24, 2009 8:38 PM writes...

The problem isn't that people produce biased studies; it's that it's so costly and difficult to catch their biases. In economics and public policy there are biased studies all the time--"publicly fund my stadium and we will create 10,000 jobs" being a common example--but it is usually very easy to look at the study's data, design, and interpretation to show what is wrong with it. Much harder to check up on a clinical trial, especially if the raw data are not available.

Permalink to Comment

15. Trackback - Cheap Internation Call >> How to make cheap international call on November 19, 2009 4:59 PM writes...

,>..] is another must read source of tips on this subject,>..]

Permalink to Comment

16. Greg Goldmacher on August 2, 2010 7:30 AM writes...

"Would it be possible to 'outsource' trials to some form of independent third party? I guess it would have to be paid for its services up front to minimize the room for economic pressure. But there'd presumably still be the threat of going elsewhere the next time, if a given trial doesn't give the desired result."

There is outsourcing of trials to contract research organizations (CROs). I work for one. We're paid regardless of the success of the drug, with our "success" measures based on the quality of the data and the smoothness of the trial process itself. We handle only one part of a trial, and we never break the blind, so there is no possibility or opportunity for us to cheat. Moreover, we're audited constantly by both the pharma companies that hire us and by the FDA.

Permalink to Comment


Remember Me?


Email this entry to:

Your email address:

Message (optional):

The Last Post
The GSK Layoffs Continue, By Proxy
The Move is Nigh
Another Alzheimer's IPO
Cutbacks at C&E News
Sanofi Pays to Get Back Into Oncology
An Irresponsible Statement About Curing Cancer
Oliver Sacks on Turning Back to Chemistry