Corante

About this Author
DBL%20Hendrix%20small.png College chemistry, 1983

Derek Lowe The 2002 Model

Dbl%20new%20portrait%20B%26W.png After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases. To contact Derek email him directly: derekb.lowe@gmail.com Twitter: Dereklowe

Chemistry and Drug Data: Drugbank
Emolecules
ChemSpider
Chempedia Lab
Synthetic Pages
Organic Chemistry Portal
PubChem
Not Voodoo
DailyMed
Druglib
Clinicaltrials.gov

Chemistry and Pharma Blogs:
Org Prep Daily
The Haystack
Kilomentor
A New Merck, Reviewed
Liberal Arts Chemistry
Electron Pusher
All Things Metathesis
C&E News Blogs
Chemiotics II
Chemical Space
Noel O'Blog
In Vivo Blog
Terra Sigilatta
BBSRC/Douglas Kell
ChemBark
Realizations in Biostatistics
Chemjobber
Pharmalot
ChemSpider Blog
Pharmagossip
Med-Chemist
Organic Chem - Education & Industry
Pharma Strategy Blog
No Name No Slogan
Practical Fragments
SimBioSys
The Curious Wavefunction
Natural Product Man
Fragment Literature
Chemistry World Blog
Synthetic Nature
Chemistry Blog
Synthesizing Ideas
Business|Bytes|Genes|Molecules
Eye on FDA
Chemical Forums
Depth-First
Symyx Blog
Sceptical Chymist
Lamentations on Chemistry
Computational Organic Chemistry
Mining Drugs
Henry Rzepa


Science Blogs and News:
Bad Science
The Loom
Uncertain Principles
Fierce Biotech
Blogs for Industry
Omics! Omics!
Young Female Scientist
Notional Slurry
Nobel Intent
SciTech Daily
Science Blog
FuturePundit
Aetiology
Gene Expression (I)
Gene Expression (II)
Sciencebase
Pharyngula
Adventures in Ethics and Science
Transterrestrial Musings
Slashdot Science
Cosmic Variance
Biology News Net


Medical Blogs
DB's Medical Rants
Science-Based Medicine
GruntDoc
Respectful Insolence
Diabetes Mine


Economics and Business
Marginal Revolution
The Volokh Conspiracy
Knowledge Problem


Politics / Current Events
Virginia Postrel
Instapundit
Belmont Club
Mickey Kaus


Belles Lettres
Uncouth Reflections
Arts and Letters Daily
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline

In the Pipeline

« Merck and J&J Finally Come to Terms | Main | Return of the Magic Methyl Group »

April 20, 2011

Nothing Personal

Email This Entry

Posted by Derek

There's an interesting follow-up over at SciBX to Bruce Booth's piece on the reproducibility of academic research. Booth, in his position as a venture capital purse-string holder, advocated caution and careful verification of exciting academic discoveries before starting the company-formation process.

The SciBX folks followed up with him and with several other VCs. Booth sticks to his position, and says that his firm, Atlas Venture, has allocated money to allow CROs to do reality checks on the new ideas that they see. Daphne Zohar at PureTech Ventures takes a similar line, but says that they do this sort of work with the originators of the technology, giving it a quiet shakedown before talking to investors. They do use CROs when appropriate, though.

On the other end of the spectrum, though, you have Camille Samuels at Versant Ventures:

“I think the best way to prevent yourself from funding biotechs that have a faulty scientific basis is to develop a trusting relationship with the scientific founders,” she told SciBX. “I think that starting a productive, long-term business relationship is hard to do if you use a ‘guilty before proven innocent’ approach.”

Samuels favors vetting the science with a top-notch scientific advisory team before launching a company. “If you hire great scientists to the company you will uncover the ‘over-reaching’ before you’ve spent any real money,” she noted.

I'm not so sure about that myself. While I agree that a good relationship between the VC people and the founding scientists is crucial, I think that any such relationship worthy of the name should be able to stand up to this sort of review. Everyone involved should be wise enough to realize this, and not take it personally. "Guilty until proven innocent", after all, is not such a bad attitude when you're looking at something that's interesting enough to trigger millions of dollars worth of investment. If the idea or technology is strong enough for real money, it's strong enough to handle a good shaking - and if it isn't, you'd want to know that as early as possible.

And to be honest, isn't it the same attitude that greets any big new discovery when it hits the literature? When some hot news comes out in a competitive field, the first thought of all the outside teams is "I wonder if that's real?" A big name or a trusted institution will buy a bit more benefit of the doubt, but not much, as well it shouldn't. I'm willing to believe that interesting results from a reliable research group are probably true, but I'll only put them in the "solid" category when I've seen someone else reproduce them (or have done it myself). That's science.

Comments (32) + TrackBacks (0) | Category: Academia (vs. Industry) | Business and Markets


COMMENTS

1. emjeff on April 20, 2011 8:00 AM writes...

Completely agree. Extraordinary claims require extraordinary proof.

Permalink to Comment

2. newnickname on April 20, 2011 8:07 AM writes...

"Samuels favors vetting the science with a top-notch scientific advisory team before launching a company." Although he may have been overly self-effacing in some of his talks, Whitesides jokes about his academic consulting group advising the founders of Genzyme not to pursue Ceredase or enzyme replacement therapy. The biotechs where I worked had a pattern of engaging highly paid "experts" which is where I learned the following definition of the term: ex-pert (noun) - someone who calls themself an expert.

In the history of biotech, there are also times when VC should probably have taken the biotech's self generated claims and had them vetted for reproducibility by an outside third party.

Permalink to Comment

3. p on April 20, 2011 8:15 AM writes...

Any scientist should, at any time, be willing to demonstrate how they got their results and that they're reproducible.* If a "scientist" approaches you with a brilliant idea but won't share any information or isn't willing to repeat the experiment, walk away, swiftly.

* Obviously, one has to protect intellectual rights but I'm assuming, as this is a business deal, that lawyers are involved.

Permalink to Comment

4. Will on April 20, 2011 8:22 AM writes...

From Camille:

“I think the best way to prevent yourself from funding biotechs that have a faulty scientific basis is to develop a trusting relationship with the scientific founders,”

I can see it now (adopted from conversations I have with my first grader when she tells me her teacher says it's okay to bring candy to school) :

Mr. VC - "Prof X, we are very excited with your latest discovery, and are ready to invest $20 million dollars"

Prof X - "Great, let's get going!"

Mr. VC - "Now, before we begin, I have to ask, do you really believe that your data supports your conclusions and is reproducible?"

Prof X - looks at feet

Mr. VC - "Prof X, is there something you want to tell me?"

Prof X - "yeah, sure it'll work"

Mr. VC - "Now, if I ask the people around the lab, are they going to say that also?"

Prof X - shuffles feet, continues looking down "I dunno, maybe"

MR. VC - "Prof X..."

Prof X - "Oh, alright, we probably don't know for sure if it'll work." Shuffles off to retract articles from Nature and Cell and withdraw patent applications

Permalink to Comment

5. Jim on April 20, 2011 8:25 AM writes...

Isn't "Guilty until proven innocent" just an accusatory rewording of "Nullis in verba"?

Permalink to Comment

6. johnnyboy on April 20, 2011 8:40 AM writes...

totally agree. Samuels' argument is complete BS. Science shouldn't be about 'relationships', it should be about data. 'Relationships' are the main reason millions of dollars are thrown at bad ideas. And 'guilty until proven innocent' is the approach that I was taught scientists should have towards their own data - when your experiment points towards a certain hypothesis, you should try to devise other experiments to disprove that hypothesis, before you get anywhere near believing it. That's the theory anyway, i don't know if anyone really does that anymore (or ever has).

Permalink to Comment

7. MoMo on April 20, 2011 8:40 AM writes...

Every VC I ever met put the science through the ringer before they invested and were used to dealing with drug discovery charlatans. Then when not convinced they brought in the brightest and most seasoned scientists for a further squeezing.

But of course they loved the molecules they were shown. Unfortunately VCs also deal with a glut of biologists and the never-ending array of innovations that are purely diversional away from a therapeutic agent.

Permalink to Comment

8. Bryan on April 20, 2011 9:12 AM writes...

My experience with 'really big' discoveries is that these are replicated by independent labs within (at most) a couple of months of the initial report in a high profile journal and reported at meetings and in the scientific literature.

The corollary is that if you don't see replications in the literature within a year or so of the 'major breakthrough' it is likely non-replicable.

The alternative is that the 'really big' discovery was was 'too boring' to bother replicating.

Either way, if there are no replications in short order.....be wary.

Permalink to Comment

9. Hap on April 20, 2011 9:21 AM writes...

Isn't the reliance on relationships rather than data how scam artists (and other criminals) work? Just because you have a relationship with someone doesn't mean that they value it in the same way you do - they may think of you as a friend, a meal ticket, or a mark, and you probably won't know which. When money is at stake, even real friendships change.

It seems like for both professional and personal reasons, injecting relationships into VC investing seems like a bad idea.

Permalink to Comment

10. john on April 20, 2011 9:29 AM writes...

Isn't this just tip toeing around the issue of the peer review system not working very well?

Permalink to Comment

11. Hap on April 20, 2011 9:53 AM writes...

It's not the job of reviewers or journals to check that work is reproducible - it is part of peer review, but not of publication. The job of the people that review is to make sure that the work is internally consistent, that the results are consistent with the conclusions (and haven't left anything big out). Someone should check for duplication (plagiarism) as well, and for obvious signs of tampering with data. It isn't their job, though, to reproduce the data (other than in Org. Syn.). Even if reviewers and journals do their jobs well, they can't insure that data hasn't been cherry-picked (unless they request and check the raw data) or that other issues make the data irreproducible.

If work is inherently complex or requires skills or chemicals/bugs that are difficult for others to get (or, in total synthesis, requires so much work and funding that it's impossible), then it is going to be difficult for anyone to verify that it's reproducible - it might be a problem with peer review, but not one with any good solution (either not publish it or accept that it may be permanently provisional).

Permalink to Comment

12. HelicalZz on April 20, 2011 10:18 AM writes...

I wouldn't consider it 'guilty until proven innocent' so much as determination of robustness. Confirming in the models of others should be part of the development process, and understood as a necessary piece for continued capital raising. Great scientists will be testing both their models and limits on reproducibility anyway.

Permalink to Comment

13. Anonymous on April 20, 2011 10:24 AM writes...

I personally have been twice burned as a chemist collaborator working on projects for which the PIs themselves were mislead by sloppy science performed in their own labs by grad. students and postdocs who move on and leave their problems behind. One of the projects led to patents and major licensing talks which fizzled. Critical to the failure of both projects were subtle problems that a scientific advisory board would not have picked up without hard data. "Trusting relationships" between VCs and PIs are really meaningless if PIs themselves can be mislead. Show me the data, and even then I will be skeptical if it is a single scientist generating it with a huge vested interest in positive results.

Permalink to Comment

14. Bruce Booth on April 20, 2011 10:26 AM writes...

Derek - thanks for following the post and the SciBX piece. Totally agree with HelicalZZ - its not "guilty until proven innocent", its about validating the data early before a significant investment in team/infrastructure have been made.

Essentially the same thing Cami mentioned. I suspect the SciBX piece was trying to draw a distinction where there's wasn't much.

We work very collaboratively with our academic founders to start and build companies; treating them poorly in the beginning isn't conducive to that. But working with them to support external validation of the data just makes their work that much stronger. I've rarely had an academic complain about doing those sorts of 'seed' stage validation studies.

B

Permalink to Comment

15. Cellbio on April 20, 2011 10:59 AM writes...

Outside of the issue of reproducibility or robustness is the need to translate academic "promise" to commercial reality within the timeframe and budget of investors. I think most translational failings come from a difference in this framework. Yes, the work of Prof X is making fundamental contributions that will unravel the workings ofnhuman disease and yield cures, someday. That type of thinking, supported with grant dollars, does not have the same requirement for timely exploration of commercial value. Add in publication bias for wow findings, and the buyer better be careful.

So, in my opinion, and some experience, the only way is to develop a strong relationship and good dialogue that works to close the gaps so the academic founder appreciates the different environment of smallco. Doesn't always work( I am living in a company where this is our biggest challenge so feel it everyday), which is probably one reason why VCs like serial entrepreneurs. Certain universities and surrounding cultures also cultivate this idea, hence the productivity of biotech clusters. Relationships matter a lot. We are social beasts, even us scientists.

Permalink to Comment

16. john on April 20, 2011 11:02 AM writes...

Hap
I agree that the present model of peer review can't really solve this problem. I guess that might be the point, I don't know that I have a solution for this problem.

Hate it when I end up sounding like a politician, I can identify the problems but not tell you how to fix it.

Permalink to Comment

17. TX Raven on April 20, 2011 11:30 AM writes...

The good old "trust but verify".

Permalink to Comment

18. Andy on April 20, 2011 11:32 AM writes...

Ronald Reagan: "Trust, but verify."

Due diligence should be an expected part of the relationship.

Permalink to Comment

19. TX Raven on April 20, 2011 11:32 AM writes...

The good old "trust but verify".

Permalink to Comment

20. Hap on April 20, 2011 12:02 PM writes...

1) Politicians generally tell you "I don't have any good solutions" when there are solutions that are painful and which no one wants to hear (for example, "I don't know how to reduce the deficit/debt" = "half of you (roughly) want spending cuts in other people's benefits and the other half (roughly) of you want increases in other people's taxes, and so I don't know what to tell you that will get me elected."). I think they just ignore problems for which there don't exist any good solutions.

2) I don't want funders to treat potential collaborators badly, but I don't think you want to depend on the relationship to tell you things that your results should be telling you instead.

Permalink to Comment

21. daen on April 20, 2011 1:02 PM writes...

I've worked for two biotech and one medical device startups over the last eight years. It's clear that the level of understanding varies enormously between investors. Certainly all of these companies have had some form of R&D department; none of their technologies have been in an immediately commercializable form, either because they needed further verification or more market data or because they have had process problems that needed working on. Obviously, the closer to commercial launch the technology is, the better, but it seems that very often the founders are at a make-or-break stage when looking for investment - they don't have the money for running further experiments or doing market research, which is why they are looking for investment. And obviously, as Bruce notes, from the VC's side it's very important to establish to as high a degree of confidence as possible that the technology will be good enough to provide sufficient upside before stumping up ready cash. But, as Camille says, trust is essential if the VCs are to be seen as partners in commercializing the technology, rather than as a convenient and gullible source of the folding stuff to fund ongoing and apparently endless R&D effort, which I have unfortunately seen (though not at my current employer, I hasten to add - Versant is one of our investors, so of course we are doing everything right ...).

Permalink to Comment

22. TJM on April 20, 2011 1:04 PM writes...

Cellbio – I agree with you that it is crucial to have a “view to the horizon” for when the idea will begin creating value. That was the most important thing I taught VC’s a decade ago when they were nibbling at biotech, but did not understand its process, risks, and lifecycle.

However there is another important dynamic straddling the “verify, verify”, and “relationships will prove out” camps: As an idea becomes more validated, the owners can demand a greater share of the value from a widening group of bidders. So, the projected returns for the VC may drop dramatically as that risk goes down. Well-versed VC's now use a range of tools (and relationships) and knowledge of the commercial landscape (including when to sell and if Big Pharma will buy) to understand the value/risk better than the innovator. Having/hiring a leading scientist in the field helps to wean out the charlatans, as other note.

This is similar to when Pharma pays a huge premium for rights to a product at the end of Phase II, with other bidding up the price to near (or above) consensus estimates of real value. Preclinical candidates are cheap but can pay off big if you understand the disease & economic opportunity far better than the innovator.


On the other hand, well organized university or regional incubator groups help to even the playing field for the innovators in a similar way by understanding and having a process to manage the same risk/value and economic landscape.

Permalink to Comment

23. Zippy on April 20, 2011 1:48 PM writes...

When evaluating important new technologies to be purchased or licensed from biotechs or instrument vendors, we generally performed a site visit before signing. Usually two types of experiments were done. First, the external company would pick their best demonstration experiment. Second, we would send reagents and instructions for an experiment that we designed and which they agreed was in scope. For technologies involving new instrumentation, a power down and reboot was done. No one ever objected to this approach, but not all passed. This process seemed useful to both parties and lead to some productive relationships.

For more complex scientific problems that could not be evaluated in real time, verification experiments were sometimes run at one or both sites. The external party would run specified experiments with blinded samples and internal research would use the use reagents and protocols from the external source to replicate their results. But, of course, you need internal science for that.

Permalink to Comment

24. Physic on April 20, 2011 3:12 PM writes...

Cold Fusion.

Permalink to Comment

25. Jack H. Pincus on April 20, 2011 3:22 PM writes...

The problem is real. Ask any academic or corporate scientist who has tried to reproduce published experiments. There are other causes of the problem besides academic bias as desribed by Bruce Booth. These include: incomplete descriptions of experimental procedures; tacit knowledge needed to reproduce experiments that authors assume is widespread in a field; academic lab heads letting their labs run on automatic pilot including with little or no supervision of graduate students, postdocs, or technicians; inadequate data review; inadequate review of lab procedures; and eschewing any form of project management or accountability for results. All of these problems are ingrained in academic research culture and are exacerbated in proporiton to the size of a lab. Sadly, the situation is not likely to change soon.

Permalink to Comment

26. RKN on April 20, 2011 3:48 PM writes...

@JackHPincus

In addition there is the Decline Effect.

tinyurl.com/2bjlacx

Permalink to Comment

27. MoMo on April 20, 2011 4:45 PM writes...

If academics massage the data too much the word CRIMINAL should be used. Especially when govt grants are involved. Never realized how many crooked tongues were out there until this blog.

You VCs should stick to real molecules.

Permalink to Comment

28. Chemjobber on April 20, 2011 7:16 PM writes...

A question for the wiser-than-I:

What do you do when you are the CRO scientist in the CRO/start-up relationship and you begin suspecting that the start-up's data and methods are overly hopeful?

Do you begin asking polite but skeptical questions (about positive and negative controls) or do you just sort of salute and say 'thanks for the money'?

Permalink to Comment

29. Cartesian on April 21, 2011 6:14 AM writes...

In the human rights of 1789 the article 9 is : As every man is presumed innocent until he has been declared guilty, if it should be considered necessary to arrest him, any undue harshness that is not required to secure his person must be severely curbed by Law. Also there is the 5th amendment of the U.S. Bill of rights: “…nor shall be compelled in any criminal case to be a witness against himself, nor be deprived of life, liberty, or property, without due process of law…”

Permalink to Comment

30. Cellbio on April 21, 2011 8:56 AM writes...

Interesting points, TJM.

My current venture is in self- incubation mode, because it is early for VCs today, and the founders feel they have been ripped off by VCs before. However, one theft was a 50MM investment that only produced tax loss credits, and the other a modest revenue stream but no exit yet. However, they are also reluctant to spend the institutes money to advance the technology to drisk and improve value for all of us. I do believe the risk tolerance and experience of VCs is under appreciated, and it becomes apparent if you think about what venture you would right your own check for, or observe the strategy of academic founders in an incubated company. I am much less optimistic about incubators than I was prior to this and another two incubator experiences. I know it can work, but now have a strong appreciation for the discipline of spinning off and beginning the amicable divorce from the founders labs.

Permalink to Comment

31. NJBiologist on April 21, 2011 11:53 AM writes...

@28 Chemjobber: Who is harmed by this excess of optimism? Is it you, via a CRO equity stake in the project/client? Is it a group of VCs? Is it the investing public? Is it the journal-reading public, via published articles? In each case, different parties seem to bear responsibility for due diligence, and different rules of due diligence seem to apply.

Yes, I'm answering a question with a question, because I don't have a good, clear answer to your question.

Permalink to Comment

32. Cartesian on April 22, 2011 3:22 AM writes...

If "guilty" can be too strong, to doubt is normal.

Permalink to Comment

POST A COMMENT




Remember Me?



EMAIL THIS ENTRY TO A FRIEND

Email this entry to:

Your email address:

Message (optional):




RELATED ENTRIES
The Worst Seminar
Conference in Basel
Messed-Up Clinical Studies: A First-Hand Report
Pharma and Ebola
Lilly Steps In for AstraZeneca's Secretase Inhibitor
Update on Alnylam (And the Direction of Things to Come)
There Must Have Been Multiple Chances to Catch This
Weirdly, Tramadol Is Not a Natural Product After All