Corante

About this Author
DBL%20Hendrix%20small.png College chemistry, 1983

Derek Lowe The 2002 Model

Dbl%20new%20portrait%20B%26W.png After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases. To contact Derek email him directly: derekb.lowe@gmail.com Twitter: Dereklowe

Chemistry and Drug Data: Drugbank
Emolecules
ChemSpider
Chempedia Lab
Synthetic Pages
Organic Chemistry Portal
PubChem
Not Voodoo
DailyMed
Druglib
Clinicaltrials.gov

Chemistry and Pharma Blogs:
Org Prep Daily
The Haystack
Kilomentor
A New Merck, Reviewed
Liberal Arts Chemistry
Electron Pusher
All Things Metathesis
C&E News Blogs
Chemiotics II
Chemical Space
Noel O'Blog
In Vivo Blog
Terra Sigilatta
BBSRC/Douglas Kell
ChemBark
Realizations in Biostatistics
Chemjobber
Pharmalot
ChemSpider Blog
Pharmagossip
Med-Chemist
Organic Chem - Education & Industry
Pharma Strategy Blog
No Name No Slogan
Practical Fragments
SimBioSys
The Curious Wavefunction
Natural Product Man
Fragment Literature
Chemistry World Blog
Synthetic Nature
Chemistry Blog
Synthesizing Ideas
Business|Bytes|Genes|Molecules
Eye on FDA
Chemical Forums
Depth-First
Symyx Blog
Sceptical Chymist
Lamentations on Chemistry
Computational Organic Chemistry
Mining Drugs
Henry Rzepa


Science Blogs and News:
Bad Science
The Loom
Uncertain Principles
Fierce Biotech
Blogs for Industry
Omics! Omics!
Young Female Scientist
Notional Slurry
Nobel Intent
SciTech Daily
Science Blog
FuturePundit
Aetiology
Gene Expression (I)
Gene Expression (II)
Sciencebase
Pharyngula
Adventures in Ethics and Science
Transterrestrial Musings
Slashdot Science
Cosmic Variance
Biology News Net


Medical Blogs
DB's Medical Rants
Science-Based Medicine
GruntDoc
Respectful Insolence
Diabetes Mine


Economics and Business
Marginal Revolution
The Volokh Conspiracy
Knowledge Problem


Politics / Current Events
Virginia Postrel
Instapundit
Belmont Club
Mickey Kaus


Belles Lettres
Uncouth Reflections
Arts and Letters Daily
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline

In the Pipeline

« More on Grad School Pressures | Main | Things I Won't Work With: Azidoazide Azides, More Or Less »

January 8, 2013

Overly Honest Experimental Methods

Email This Entry

Posted by Derek

If you'd like a look under the hood of a lot of research publications, go over to Twitter and check the #OverlyHonestMethods tag. You're sure to find your own sins on display, things like: "Mostly it goes 43%, but once it went 95%. We reported the 95%." And "We used [this program] because doesn't everybody else?". How about "We used a modified version of Dr. Ididitfirst's apparatus, because we couldn't figure out how to make an exact replica" or "For details see Supp. Mat. We put as much as possible in there because it doesn't have to be written as carefully".

There are dozens of them, and more coming all the time. I'm adding a few myself, not that I would ever do anything like these, though, you understand.

Comments (27) + TrackBacks (0) | Category: General Scientific News


COMMENTS

1. Curious Wavefunction on January 8, 2013 12:01 PM writes...

We didn't report error bars because, frankly, none of us knew how to use the relevant module in Excel.

Permalink to Comment

2. RKN on January 8, 2013 1:10 PM writes...

To validate our finding we performed a western blot on three biological replicates, the second one looked the best. See figure 2.

Permalink to Comment

3. johnnyboy on January 8, 2013 1:31 PM writes...

We showed standard errors for all data instead of standard deviations, because: 1) it makes data look so much better; 2) doesn't everyone ?; 3) who actually cares/know anything about correct statistical methods ?

Permalink to Comment

4. connoiseur on January 8, 2013 1:43 PM writes...

@3

Providing your excuses wrt statistical methods in triplicate? Very good, sir!

Permalink to Comment

5. anonymous on January 8, 2013 2:31 PM writes...

@1,3 & 4

What the hell are statistical methods? I didn't go into organic chemistry to do math. Half the time I practice Voodoo to get my reactions to work.

Permalink to Comment

6. leftscienceawhileago on January 8, 2013 2:40 PM writes...

It would be more informative to just show the three observations on the graph, but the error bars make it look like we know statistics.

Permalink to Comment

7. Joe on January 8, 2013 2:49 PM writes...

My Ph.D. advisor had me use 1-tailed t-tests for all t-tests, because he got more stars to put above the data that way. By the end, I was so ready to be gone I just went with it. *sigh*

Permalink to Comment

8. bank on January 8, 2013 3:07 PM writes...

My postdoc advisor once insisted that I include a value for plasma cholesterol that was something like 50+/-60, because that was the "correct" way. Pointing out that the distribution of values wasn't "Normal" didn't help (maybe I should have used the word "Gaussian" instead...). It got published. Somewhere out there is a mouse with negative plasma cholesterol...

Permalink to Comment

9. Spike on January 8, 2013 4:00 PM writes...

@8. I once came across data from a study on the effect of fasting on a biomarker. The mean concentrations at one timepoint made it look like the authors had a more sensitive assay than we did. I asked them if they would mind sharing the data with me. They kindly did and low and behold, they had reported negative concentrations of the biomarker from their ELISA assay for a few subjects. No wonder the mean was so low. I haven't had the heart to tell them yet but I suspect that they know. Fast forward a couple of years and the PI came and gave a presentation. The graph looked familar but something was different. I couldn't quite put my finger on it. On closer inspection, all concentrations from the publication had been increased by 100 ng/mL!

Permalink to Comment

10. MDACC Student on January 8, 2013 4:01 PM writes...

We used 5 mice per cell type because that is how many fit into a cage.

Permalink to Comment

11. MDACC Student on January 8, 2013 4:02 PM writes...

"We used 5 mice per cell type because that is how many fit into a cage."

Permalink to Comment

12. paperclip on January 8, 2013 4:11 PM writes...

"The HPLC has peaks at 35 min. that we'd prefer not to think about. So here's the chromatogram from 0 to 30 min."

Permalink to Comment

13. Bruce Hamilton on January 8, 2013 4:57 PM writes...

The detection wavelength was 254nm because lower wavelengths had extra peaks.

Permalink to Comment

14. JRnonchemist on January 8, 2013 5:19 PM writes...

We rushed, skipped and handwaved through defining the system, nailing down all the work/energy inputs/outputs, and figuring out what measurements our experimental methodology needs statistically because we wanted to get on with drawing conclusions from the measurements we already had.

Permalink to Comment

15. milkshaken on January 8, 2013 6:32 PM writes...

I saw a statement "All melting points were measured incorrect" in the supporting info (the Chinese authors probably wanted to say "not corrected")

Permalink to Comment

16. oldnuke on January 8, 2013 8:42 PM writes...

@5 Ah, THAT'S why "combinatorial" chemistry is so hot.

Back in my day, we called that the (Thomas Alva) Edison method.

Permalink to Comment

17. The Ninja on January 9, 2013 12:55 AM writes...

The sample was concentrated in CDCl3 as much as possible to hide the minor peaks in the NMR which we wanted to ignore / cannot explain

Permalink to Comment

18. Spike on January 9, 2013 10:37 AM writes...

The doses were selected based upon the amount of drug that was available (the chemists couldn't make enough).

The study only evaluated dosing for 3 days because we ran out of drug.

We used a dose of 3.76 mg free base/kg because we forgot to correct for the salt factor.

Animals were dosed daily (except for weekends because we couldn't get anybody to come in to dose them.

Blood samples were collected at 0, 1, 3, 5 and 24 hours because we couldn't get anybody to stay in the lab after 4 pm.

.... and many variations on the above

Permalink to Comment

19. Spike on January 9, 2013 10:38 AM writes...

The doses were selected based upon the amount of drug that was available (the chemists couldn't make enough).

The study only evaluated dosing for 3 days because we ran out of drug.

We used a dose of 3.76 mg free base/kg because we forgot to correct for the salt factor.

Animals were dosed daily (except for weekends because we couldn't get anybody to come in to dose them.

Blood samples were collected at 0, 1, 3, 5 and 24 hours because we couldn't get anybody to stay in the lab after 4 pm.

.... and many variations on the above

Permalink to Comment

20. anon on January 9, 2013 12:18 PM writes...

Ubiquitous: The reaction was stirred for 16 hr at 22C... or from whatever time the tech went home to whenever he worked it up the next day, at whatever temp the lab was at that night.

Permalink to Comment

21. pete on January 9, 2013 2:42 PM writes...

Cell cultures were maintained in Enriched Medium and re-fed every other day --- except for 2 or maybe 3 times.

Permalink to Comment

22. Teddy Z on January 9, 2013 6:10 PM writes...

My late entry:
We processed the spectrum with a rediculous amount of linear prediction because the sample was so dilute we had to scan forever, so we undersampled in F1. Then, due to the horrible looking peakshape we applied a 5Hz exponential window function to really smooth out those peaks. Most likely the peaks I am about to rhapsodize on for the next four paragraphs are artifactual.

Permalink to Comment

23. Young Padawan on January 10, 2013 1:11 AM writes...

2-Methoxy-6-(3-( (tetrahydro-2H -pyran-2-ylfoxy)-
propy1)pyridine (10)

Lactim 9 (29.0 g, 0.1 mol) and 10% Pd-C
(2.9 g) were mixed, and cyclohexene (200 mL) was added. While
adding the cyclohexene, the reaction burst into flames. The flames
were extinguished with a CO2 fire extinguisher, and the hot flask
was cooled in an ice bath. (The fire was due to the direct mixing
of 9 and 10% Pd-C, which ignited the solvent, cyclohexene. This
step has since been modified. In subsequent reactions 10% Pd-C
and cyclohexene were premixed and heated to reflux, where upon
an exothermic reaction occurs.26 After the vigorous exotherm
subsided, the substrate was added slowly. This step has been
successfully run on 40 g of compound with no further accidents.)

From : JOC, 1986, 51 (12), 2184-2191
I wish he had specified the brand of the fire extinguisher though... Doesn't seem very scientific this way.

Permalink to Comment

24. Jon on January 10, 2013 9:29 AM writes...

I don't think the experimental from that JOC paper is overly honest. That's the sort of thing I'd expect to see in OPRD or maybe Org Syn and quite useful to know, although you could probably skip the firefighting details. Heck, OPRD reviewers these days would probably demand safety data on that step.

Permalink to Comment

25. emjeff on January 11, 2013 10:42 AM writes...

Am I the only one who thinks this could do more harm than good? #notafan

Permalink to Comment

26. Hap on January 11, 2013 2:08 PM writes...

It could be harmful if 1) people get new ideas on how to misrepresent their research (though I thought that "security through obscurity" was pretty much good for securing illusions and not much else) or 2) if people think that scientists just do what gets them paid and thus that science is socially preferred and not necessarily intrinsically useful. Neither of those consequences seems likely to me, though (for 1), publicizing the peccadillos of authors might make it easier for editors and reviewers to catch them, if they care, and might provide a socially acceptable way to pressure people to stop while for 2), it seems like there is no shortage of reasons for people to deprecate science, particularly if it tells them things they do not like).

It could also make people think that the behavior is acceptable if their sarcasm detectors are nonfunctional. Of course, if that's the case, then I wouldn't have much that I could say. Don't like that.

Permalink to Comment

27. DensityDuck on January 17, 2013 7:11 PM writes...

And, of course, there's the old standby "GLOSSARY FOR RESEARCH PAPERS: STRICTLY SPEAKING".

Permalink to Comment

POST A COMMENT




Remember Me?



EMAIL THIS ENTRY TO A FRIEND

Email this entry to:

Your email address:

Message (optional):




RELATED ENTRIES
XKCD on Protein Folding
The 2014 Chemistry Nobel: Beating the Diffraction Limit
German Pharma, Or What's Left of It
Sunesis Fails with Vosaroxin
A New Way to Estimate a Compound's Chances?
Meinwald Honored
Molecular Biology Turns Into Chemistry
Speaking at Northeastern