"You like those scatterplots, don't you?", someone said to me the other day. And I can't deny it. On most projects that my lab has been assigned to, at some point I end up messing around with all the project data, plotting one thing against another and looking for correlations.
Often what I find is negative. Plotting liver microsome stability (a measure, in theory, of one of the major pathways for drug metabolism) against compound blood levels in animal dosing has rarely, in my perhaps unrepresentative experience, shown much of a correlation. In vivo blood levels are just too complicated, and influenced by too many other things. But I'm often surprised by how many people assume that there's a correlation - because, to a first approximation, it sort of makes sense that there might be - without actually having run the numbers.
That's a theme that keeps recurring: a fair amount of what people think they know about their project isn't true. I think it's because we keep reaching for simple explanations and rules of thumb, in hopes that we can get some sort of grip on the data. We give these too much weight, though, especially if we don't examine them every so often to see if they're still holding up (or if they ever did in the first place).
Another factor is good ol' fear. It's unnerving to face up to the fact that you don't know why your compounds are behaving the way that they are, and that you don't know what to do about it. It's no fun to plot your primary assay data against your secondary data and see a dropped-paintcan scatter instead of a correlation, because that kind of thing can set your whole project back months (or kill it altogether). One of the biggest problems in an information-driven field is that not everyone wants to know.
One time when I was giving the numbers a complete run-through, I noticed one of the plots actually seemed to have a fairly good shape to it. Y-axis was potency (plotted as -log), and there it was, actually increasing - broadly, messily, but undeniably - with the X-axis, which was. . .corporate compound number, the one assigned to each new compound as it was sent in for the assay. Oh, well. It showed that we were making progress, anyway. And at least nobody suggested that we attempt to give the compounds numbers from years in the future, in order to make them instant surefire winners. I've heard sillier suggestions.
1. John Johnson on October 23, 2006 4:57 AM writes...
This is one of the things that drives me nuts about the business side of drug development. Every blip in the data is given so much weight and overinterpreted to the point of meaninglessness.
But then again, from a venture capitalist's perspective, you predict with the data you have.
Permalink to Comment2. Kay on October 23, 2006 6:11 AM writes...
I have been intrigued for some years now by the willingness of medicinal chemists to implement handwaving-in-overdrive rules and methods. I am fascinated by this because the outcome is not academic, but rather the outcome is the financial survival of their employers (and public health). It's OK to admit that we know little and we guess often. To Derek's credit, he does this often and he seems not to be labeled a 'heretic' or similar.
Is there some sense that the business guys would fire us all if they learned that things were a little less scientific than advertised?
Permalink to Comment3. milkshake on October 23, 2006 8:18 AM writes...
I had too many meetings with people saying "we have to have all compounds with microsome half-time at least 45 min" or "this compound is bad because it has fast metabolism in microsomes" or "we cannot develop this compound because it has a hydroxylated/de-methylated metabolite that is likely to be active"
I did not make myself too popular by pointing out that they did not even run control compounds in their microsome assay and that perhaps half of the drugs currently on market would not pass their criterions for selecting preclinical candidates. It was all basicaly a wishful thinking and management cover-my-ass balooney.
Permalink to Comment4. tom bartlett on October 23, 2006 8:42 AM writes...
"This is one of the things that drives me nuts about the business side of drug development. Every blip in the data is given so much weight and overinterpreted to the point of meaninglessness."
To paraphrase Rumsfeld, "you go to war with the data you have, not the data you want." I have to decide what to make THIS week; not after the biologists have beaten my analog from last week to death with every assay known to man.
Permalink to Comment5. Chrispy on October 23, 2006 4:24 PM writes...
Permalink to CommentIn my House of Magic Compounds they developed a hormone drug which ended up going into people before the chimp data was in. It's a good thing, too, because the drug acted as expected in people but did nothing in chimps. Had it happened in the reverse order it would have been very hard to move into people...
6. Paul Dietz on October 24, 2006 10:25 AM writes...
In his autobiography, Stanislaw Ulam related a story of a post-war seminar at Los Alamos. The presenter had put up a slide with a cloud of widely scattered points and had rather optimistically drawn a line through them. John von Neumann leaned over to Ulam and whispered, 'well, at least they lie on a plane'.
Permalink to Comment7. srp on October 24, 2006 4:25 PM writes...
There are other businesses where cause and effect relationships are hard to understand (e.g. direct mail or semiconductor manufacturing). Successful firms in these areas do not try to develop intuitively plausible but unlikely-to-be-accurate intermediate theories ( e.g. if x goes up y should go up).
Instead they do a variety of acausal hill-climbing experiments (e.g., what if we mail out a small test batch of this text with this picture on the left and that one on the right?) and see what happens. Once something is gotten to work (e.g. 1% respond to the mailing), it is kept exactly the same for any production use.
Why couldn't drug firms simply admit to themselves that cause and effect is too complicated and variable to tease out, and stick to multifactor experimentation? Relying on poor intermediate causal measures is adjusting a control variable to noise instead of signal.
Permalink to Comment