Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases.
To contact Derek email him directly: derekb.lowe@gmail.com
Twitter: Dereklowe
If you'd like a look under the hood of a lot of research publications, go over to Twitter and check the #OverlyHonestMethods tag. You're sure to find your own sins on display, things like: "Mostly it goes 43%, but once it went 95%. We reported the 95%." And "We used [this program] because doesn't everybody else?". How about "We used a modified version of Dr. Ididitfirst's apparatus, because we couldn't figure out how to make an exact replica" or "For details see Supp. Mat. We put as much as possible in there because it doesn't have to be written as carefully".
There are dozens of them, and more coming all the time. I'm adding a few myself, not that I would ever do anything like these, though, you understand.
We showed standard errors for all data instead of standard deviations, because: 1) it makes data look so much better; 2) doesn't everyone ?; 3) who actually cares/know anything about correct statistical methods ?
My Ph.D. advisor had me use 1-tailed t-tests for all t-tests, because he got more stars to put above the data that way. By the end, I was so ready to be gone I just went with it. *sigh*
My postdoc advisor once insisted that I include a value for plasma cholesterol that was something like 50+/-60, because that was the "correct" way. Pointing out that the distribution of values wasn't "Normal" didn't help (maybe I should have used the word "Gaussian" instead...). It got published. Somewhere out there is a mouse with negative plasma cholesterol...
@8. I once came across data from a study on the effect of fasting on a biomarker. The mean concentrations at one timepoint made it look like the authors had a more sensitive assay than we did. I asked them if they would mind sharing the data with me. They kindly did and low and behold, they had reported negative concentrations of the biomarker from their ELISA assay for a few subjects. No wonder the mean was so low. I haven't had the heart to tell them yet but I suspect that they know. Fast forward a couple of years and the PI came and gave a presentation. The graph looked familar but something was different. I couldn't quite put my finger on it. On closer inspection, all concentrations from the publication had been increased by 100 ng/mL!
14. JRnonchemist on January 8, 2013 5:19 PM writes...
We rushed, skipped and handwaved through defining the system, nailing down all the work/energy inputs/outputs, and figuring out what measurements our experimental methodology needs statistically because we wanted to get on with drawing conclusions from the measurements we already had.
Ubiquitous: The reaction was stirred for 16 hr at 22C... or from whatever time the tech went home to whenever he worked it up the next day, at whatever temp the lab was at that night.
My late entry:
We processed the spectrum with a rediculous amount of linear prediction because the sample was so dilute we had to scan forever, so we undersampled in F1. Then, due to the horrible looking peakshape we applied a 5Hz exponential window function to really smooth out those peaks. Most likely the peaks I am about to rhapsodize on for the next four paragraphs are artifactual.
Lactim 9 (29.0 g, 0.1 mol) and 10% Pd-C
(2.9 g) were mixed, and cyclohexene (200 mL) was added. While
adding the cyclohexene, the reaction burst into flames. The flames
were extinguished with a CO2 fire extinguisher, and the hot flask
was cooled in an ice bath. (The fire was due to the direct mixing
of 9 and 10% Pd-C, which ignited the solvent, cyclohexene. This
step has since been modified. In subsequent reactions 10% Pd-C
and cyclohexene were premixed and heated to reflux, where upon
an exothermic reaction occurs.26 After the vigorous exotherm
subsided, the substrate was added slowly. This step has been
successfully run on 40 g of compound with no further accidents.)
From : JOC, 1986, 51 (12), 2184-2191
I wish he had specified the brand of the fire extinguisher though... Doesn't seem very scientific this way.
I don't think the experimental from that JOC paper is overly honest. That's the sort of thing I'd expect to see in OPRD or maybe Org Syn and quite useful to know, although you could probably skip the firefighting details. Heck, OPRD reviewers these days would probably demand safety data on that step.
It could be harmful if 1) people get new ideas on how to misrepresent their research (though I thought that "security through obscurity" was pretty much good for securing illusions and not much else) or 2) if people think that scientists just do what gets them paid and thus that science is socially preferred and not necessarily intrinsically useful. Neither of those consequences seems likely to me, though (for 1), publicizing the peccadillos of authors might make it easier for editors and reviewers to catch them, if they care, and might provide a socially acceptable way to pressure people to stop while for 2), it seems like there is no shortage of reasons for people to deprecate science, particularly if it tells them things they do not like).
It could also make people think that the behavior is acceptable if their sarcasm detectors are nonfunctional. Of course, if that's the case, then I wouldn't have much that I could say. Don't like that.
1. Curious Wavefunction on January 8, 2013 12:01 PM writes...
We didn't report error bars because, frankly, none of us knew how to use the relevant module in Excel.
Permalink to Comment2. RKN on January 8, 2013 1:10 PM writes...
To validate our finding we performed a western blot on three biological replicates, the second one looked the best. See figure 2.
Permalink to Comment3. johnnyboy on January 8, 2013 1:31 PM writes...
We showed standard errors for all data instead of standard deviations, because: 1) it makes data look so much better; 2) doesn't everyone ?; 3) who actually cares/know anything about correct statistical methods ?
Permalink to Comment4. connoiseur on January 8, 2013 1:43 PM writes...
@3
Providing your excuses wrt statistical methods in triplicate? Very good, sir!
Permalink to Comment5. anonymous on January 8, 2013 2:31 PM writes...
@1,3 & 4
What the hell are statistical methods? I didn't go into organic chemistry to do math. Half the time I practice Voodoo to get my reactions to work.
Permalink to Comment6. leftscienceawhileago on January 8, 2013 2:40 PM writes...
It would be more informative to just show the three observations on the graph, but the error bars make it look like we know statistics.
Permalink to Comment7. Joe on January 8, 2013 2:49 PM writes...
My Ph.D. advisor had me use 1-tailed t-tests for all t-tests, because he got more stars to put above the data that way. By the end, I was so ready to be gone I just went with it. *sigh*
Permalink to Comment8. bank on January 8, 2013 3:07 PM writes...
My postdoc advisor once insisted that I include a value for plasma cholesterol that was something like 50+/-60, because that was the "correct" way. Pointing out that the distribution of values wasn't "Normal" didn't help (maybe I should have used the word "Gaussian" instead...). It got published. Somewhere out there is a mouse with negative plasma cholesterol...
Permalink to Comment9. Spike on January 8, 2013 4:00 PM writes...
@8. I once came across data from a study on the effect of fasting on a biomarker. The mean concentrations at one timepoint made it look like the authors had a more sensitive assay than we did. I asked them if they would mind sharing the data with me. They kindly did and low and behold, they had reported negative concentrations of the biomarker from their ELISA assay for a few subjects. No wonder the mean was so low. I haven't had the heart to tell them yet but I suspect that they know. Fast forward a couple of years and the PI came and gave a presentation. The graph looked familar but something was different. I couldn't quite put my finger on it. On closer inspection, all concentrations from the publication had been increased by 100 ng/mL!
Permalink to Comment10. MDACC Student on January 8, 2013 4:01 PM writes...
We used 5 mice per cell type because that is how many fit into a cage.
Permalink to Comment11. MDACC Student on January 8, 2013 4:02 PM writes...
"We used 5 mice per cell type because that is how many fit into a cage."
Permalink to Comment12. paperclip on January 8, 2013 4:11 PM writes...
"The HPLC has peaks at 35 min. that we'd prefer not to think about. So here's the chromatogram from 0 to 30 min."
Permalink to Comment13. Bruce Hamilton on January 8, 2013 4:57 PM writes...
The detection wavelength was 254nm because lower wavelengths had extra peaks.
Permalink to Comment14. JRnonchemist on January 8, 2013 5:19 PM writes...
We rushed, skipped and handwaved through defining the system, nailing down all the work/energy inputs/outputs, and figuring out what measurements our experimental methodology needs statistically because we wanted to get on with drawing conclusions from the measurements we already had.
Permalink to Comment15. milkshaken on January 8, 2013 6:32 PM writes...
I saw a statement "All melting points were measured incorrect" in the supporting info (the Chinese authors probably wanted to say "not corrected")
Permalink to Comment16. oldnuke on January 8, 2013 8:42 PM writes...
@5 Ah, THAT'S why "combinatorial" chemistry is so hot.
Back in my day, we called that the (Thomas Alva) Edison method.
Permalink to Comment17. The Ninja on January 9, 2013 12:55 AM writes...
The sample was concentrated in CDCl3 as much as possible to hide the minor peaks in the NMR which we wanted to ignore / cannot explain
Permalink to Comment18. Spike on January 9, 2013 10:37 AM writes...
The doses were selected based upon the amount of drug that was available (the chemists couldn't make enough).
The study only evaluated dosing for 3 days because we ran out of drug.
We used a dose of 3.76 mg free base/kg because we forgot to correct for the salt factor.
Animals were dosed daily (except for weekends because we couldn't get anybody to come in to dose them.
Blood samples were collected at 0, 1, 3, 5 and 24 hours because we couldn't get anybody to stay in the lab after 4 pm.
.... and many variations on the above
Permalink to Comment19. Spike on January 9, 2013 10:38 AM writes...
The doses were selected based upon the amount of drug that was available (the chemists couldn't make enough).
The study only evaluated dosing for 3 days because we ran out of drug.
We used a dose of 3.76 mg free base/kg because we forgot to correct for the salt factor.
Animals were dosed daily (except for weekends because we couldn't get anybody to come in to dose them.
Blood samples were collected at 0, 1, 3, 5 and 24 hours because we couldn't get anybody to stay in the lab after 4 pm.
.... and many variations on the above
Permalink to Comment20. anon on January 9, 2013 12:18 PM writes...
Ubiquitous: The reaction was stirred for 16 hr at 22C... or from whatever time the tech went home to whenever he worked it up the next day, at whatever temp the lab was at that night.
Permalink to Comment21. pete on January 9, 2013 2:42 PM writes...
Cell cultures were maintained in Enriched Medium and re-fed every other day --- except for 2 or maybe 3 times.
Permalink to Comment22. Teddy Z on January 9, 2013 6:10 PM writes...
My late entry:
Permalink to CommentWe processed the spectrum with a rediculous amount of linear prediction because the sample was so dilute we had to scan forever, so we undersampled in F1. Then, due to the horrible looking peakshape we applied a 5Hz exponential window function to really smooth out those peaks. Most likely the peaks I am about to rhapsodize on for the next four paragraphs are artifactual.
23. Young Padawan on January 10, 2013 1:11 AM writes...
2-Methoxy-6-(3-( (tetrahydro-2H -pyran-2-ylfoxy)-
propy1)pyridine (10)
Lactim 9 (29.0 g, 0.1 mol) and 10% Pd-C
(2.9 g) were mixed, and cyclohexene (200 mL) was added. While
adding the cyclohexene, the reaction burst into flames. The flames
were extinguished with a CO2 fire extinguisher, and the hot flask
was cooled in an ice bath. (The fire was due to the direct mixing
of 9 and 10% Pd-C, which ignited the solvent, cyclohexene. This
step has since been modified. In subsequent reactions 10% Pd-C
and cyclohexene were premixed and heated to reflux, where upon
an exothermic reaction occurs.26 After the vigorous exotherm
subsided, the substrate was added slowly. This step has been
successfully run on 40 g of compound with no further accidents.)
From : JOC, 1986, 51 (12), 2184-2191
Permalink to CommentI wish he had specified the brand of the fire extinguisher though... Doesn't seem very scientific this way.
24. Jon on January 10, 2013 9:29 AM writes...
I don't think the experimental from that JOC paper is overly honest. That's the sort of thing I'd expect to see in OPRD or maybe Org Syn and quite useful to know, although you could probably skip the firefighting details. Heck, OPRD reviewers these days would probably demand safety data on that step.
Permalink to Comment25. emjeff on January 11, 2013 10:42 AM writes...
Am I the only one who thinks this could do more harm than good? #notafan
Permalink to Comment26. Hap on January 11, 2013 2:08 PM writes...
It could be harmful if 1) people get new ideas on how to misrepresent their research (though I thought that "security through obscurity" was pretty much good for securing illusions and not much else) or 2) if people think that scientists just do what gets them paid and thus that science is socially preferred and not necessarily intrinsically useful. Neither of those consequences seems likely to me, though (for 1), publicizing the peccadillos of authors might make it easier for editors and reviewers to catch them, if they care, and might provide a socially acceptable way to pressure people to stop while for 2), it seems like there is no shortage of reasons for people to deprecate science, particularly if it tells them things they do not like).
It could also make people think that the behavior is acceptable if their sarcasm detectors are nonfunctional. Of course, if that's the case, then I wouldn't have much that I could say. Don't like that.
Permalink to Comment27. DensityDuck on January 17, 2013 7:11 PM writes...
And, of course, there's the old standby "GLOSSARY FOR RESEARCH PAPERS: STRICTLY SPEAKING".
Permalink to Comment