About this Author
DBL%20Hendrix%20small.png College chemistry, 1983

Derek Lowe The 2002 Model

Dbl%20new%20portrait%20B%26W.png After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases. To contact Derek email him directly: Twitter: Dereklowe

Chemistry and Drug Data: Drugbank
Chempedia Lab
Synthetic Pages
Organic Chemistry Portal
Not Voodoo

Chemistry and Pharma Blogs:
Org Prep Daily
The Haystack
A New Merck, Reviewed
Liberal Arts Chemistry
Electron Pusher
All Things Metathesis
C&E News Blogs
Chemiotics II
Chemical Space
Noel O'Blog
In Vivo Blog
Terra Sigilatta
BBSRC/Douglas Kell
Realizations in Biostatistics
ChemSpider Blog
Organic Chem - Education & Industry
Pharma Strategy Blog
No Name No Slogan
Practical Fragments
The Curious Wavefunction
Natural Product Man
Fragment Literature
Chemistry World Blog
Synthetic Nature
Chemistry Blog
Synthesizing Ideas
Eye on FDA
Chemical Forums
Symyx Blog
Sceptical Chymist
Lamentations on Chemistry
Computational Organic Chemistry
Mining Drugs
Henry Rzepa

Science Blogs and News:
Bad Science
The Loom
Uncertain Principles
Fierce Biotech
Blogs for Industry
Omics! Omics!
Young Female Scientist
Notional Slurry
Nobel Intent
SciTech Daily
Science Blog
Gene Expression (I)
Gene Expression (II)
Adventures in Ethics and Science
Transterrestrial Musings
Slashdot Science
Cosmic Variance
Biology News Net

Medical Blogs
DB's Medical Rants
Science-Based Medicine
Respectful Insolence
Diabetes Mine

Economics and Business
Marginal Revolution
The Volokh Conspiracy
Knowledge Problem

Politics / Current Events
Virginia Postrel
Belmont Club
Mickey Kaus

Belles Lettres
Uncouth Reflections
Arts and Letters Daily
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline

In the Pipeline

« Mannkind's Latest Data | Main | High Throughput Screening Services »

August 16, 2013

An HIV Structure Breakthrough? Or "Complete Rubbish"?

Email This Entry

Posted by Derek

Structural biology needs no introduction for people doing drug discovery. This wasn't always so. Drugs were discovered back in the days when people used to argue about whether those "receptor" thingies were real objects (as opposed to useful conceptual shorthand), and before anyone had any idea of what an enzyme's active site might look like. And even today, there are targets, and whole classes of targets, for which we can't get enough structural information to help us out much.

But when you can get it, structure can be a wonderful thing. X-ray crystallography of proteins, and protein-ligand complexes has revealed so much useful information that it's hard to know where to start. It's not the magic wand - you can't look at an empty binding site and just design something right at your desk that'll be a potent ligand right off the bat. And you can't look at a series of ligand-bound structures and say which one is the most potent, not in most situations, anyway. But you still learn things from X-ray structures that you could never have known otherwise.

It's not the only game in town, either. NMR structures are very useful, although the X-ray ones can be easier to get, especially in these days of automated synchroton beamlines and powerful number-crunching. But what if your protein doesn't crystallize? And what if there are things happening in solution that you'd never pick up on from the crystallized form? You're not going to watch your protein rearrange into a new ligand-bound conformation with X-ray crystallography, that's for sure. No, even though NMR structures can be a pain to get, and have to be carefully interpreted, they'll also show you things you'd never had seen.

And there are more exotic methods. Earlier this summer, there was a startling report of a structure of the HIV surface proteins gp120 and gp41 obtained through cryogenic electron microscopy. This is a very important and very challenging field to work in. What you've got there is a membrane-bound protein-protein interaction, which is just the sort of thing that the other major structure-determination techniques can't handle well. At the same time, though, the number of important proteins involved in this sort of thing is almost beyond listing. Cryo-EM, since it observes the native proteins in their natural environment, without tags or stains, has a lot of potential, but it's been extremely hard to get the sort of resolution with it that's needed on such targets.

Joseph Sodroski's group at Harvard, longtime workers in this area, published their 6-angstrom-resolution structure of the protein complex in PNAS. But according to this new article in Science, the work has been an absolute lightning rod ever since it appeared. Many other structural biologists think that the paper is so flawed that it never should have seen print. No, I'm not exaggerating:

Several respected HIV/AIDS researchers are wowed by the work. But others—structural biologists in particular—assert that the paper is too good to be true and is more likely fantasy than fantastic. "That paper is complete rubbish," charges Richard Henderson, an electron microscopy pioneer at the MRC Laboratory of Molecular Biology in Cambridge, U.K. "It has no redeeming features whatsoever."

. . .Most of the structural biologists and HIV/AIDS researchers Science spoke with, including several reviewers, did not want to speak on the record because of their close relations with Sodroski or fear that they'd be seen as competitors griping—and some indeed are competitors. Two main criticisms emerged. Structural biologists are convinced that Sodroski's group, for technical reasons, could not have obtained a 6-Å resolution structure with the type of microscope they used. The second concern is even more disturbing: They solved the structure of a phantom molecule, not the trimer.

Cryo-EM is an art form. You have to freeze your samples in an aqueous system, but without making ice. The crystals of normal ice formation will do unsightly things to biological samples, on both the macro and micro levels, so you have to form "vitreous ice", a glassy amorphous form of frozen water, which is odd enough that until the 1980s many people considered it impossible. Once you've got your protein particles in this matrix, though, you can't just blast away at full power with your electron beam, because that will also tear things up. You have to take a huge number of runs at lower power, and analyze them through statistical techniques. The Sodolski HIV structure, for example, is the product of 670,000 single-particle images.

But its critics say that it's also the product of wishful thinking.:

The essential problem, they contend, is that Sodroski and Mao "aligned" their trimers to lower-resolution images published before, aiming to refine what was known. This is a popular cryo-EM technique but requires convincing evidence that the particles are there in the first place and rigorous tests to ensure that any improvements are real and not the result of simply finding a spurious agreement with random noise. "They should have done lots of controls that they didn't do," (Sriram) Subramaniam asserts. In an oft-cited experiment that aligns 1000 computer-generated images of white noise to a picture of Albert Einstein sticking out his tongue, the resulting image still clearly shows the famous physicist. "You get a beautiful picture of Albert Einstein out of nothing," Henderson says. "That's exactly what Sodroski and Mao have done. They've taken a previously published structure and put atoms in and gone down into a hole." Sodroski and Mao declined to address specific criticisms about their studies.

Well, they decline to answer them in response to a news item in Science. They've indicated a willingness to take on all comers in the peer-reviewed literature, but otherwise, in print, they're doing the we-stand-by-our-results-no-comment thing. Sodroski himself, with his level of experience in the field, seems ready to defend this paper vigorously, but there seem to be plenty of others willing to attack. We'll have to see how this plays out in the coming months - I'll update as things develop.

Comments (34) + TrackBacks (0) | Category: Analytical Chemistry | Biological News | In Silico | Infectious Diseases


1. Anonymous on August 16, 2013 10:00 AM writes...

Even X-ray crystallography is subject to over-fitting data, therefore one has to keep a portion of data out of the fitting algorithms to assess the real level of fit. Similarly, this group should have used only a partial model, and then look what happens outside the model to see if there is anything more than white noise.

Permalink to Comment

2. MDACC Grad on August 16, 2013 10:07 AM writes...

This is one of those situations where it would be great if the investigator could invite others out to their lab and jointly workout what they did and how it was accomplished (...assuming the PI is familiar with how his postdocs/grad students accomplished this). But grant funding and competition are going to restrict this to political-like arguments. I really doubt anyone will be discussing hard science (specific methods, calculations, buffers, etc.) in these correspondence.

Permalink to Comment

3. ptm on August 16, 2013 10:08 AM writes...

dis gon b gud

Permalink to Comment

4. ScientistSailor on August 16, 2013 10:16 AM writes...

I love a good cat fight first thing in the morning...

Permalink to Comment

5. luysii on August 16, 2013 10:27 AM writes...

I've always avoided CryoEM papers. The resemble Rohrschack tests more than anything else.

Permalink to Comment

6. PPedros on August 16, 2013 10:57 AM writes...

The sentence "Just make up a Cryo-EM" comes to mind... :D

Permalink to Comment

7. MoMo on August 16, 2013 11:04 AM writes...

Wasnt this funded by the NIH? Our tax dollars?

Its time for an Inquistion!

Permalink to Comment

8. jbosch on August 16, 2013 11:06 AM writes...

CryoEM is very powerful if done right and very helpful.
I'm linking here to my former PhD advisor:

Think of the ribosome structures for example before we had the nice high resolution Xray structures.

But you can average noise and generate your template what you used for a search. Usually you take your 1000000 particles and generate sub classes as you end up with random orientations of your protein in the vitreous ice. And then you should have a subset of images which you have not touched and verify that the subclasses identified on the first set are re-identified on the second set.

Permalink to Comment

9. Casual Observer on August 16, 2013 11:07 AM writes...

"In an oft-cited experiment that aligns 1000 computer-generated images of white noise to a picture of Albert Einstein sticking out his tongue, the resulting image still clearly shows the famous physicist."

Does anyone have a pointer to that experiment? Would really love to see it

Permalink to Comment

10. Nekekami on August 16, 2013 11:09 AM writes...

Oooh, oooh

*sells popcorn and umbrellas for the upcoming scientist feces slinging match*

Permalink to Comment

11. Toad on August 16, 2013 11:29 AM writes...

As a related aside, I'd like to reiterate to the community the point that #1 Anonymous brings up: the current state of X-ray crystallography quality.

During structure-based med chem design work, we have found a high percentage of errors and mis-assignments in the Protein Data Bank. For example, one of the most common is the reversal of the O and N atoms in the carboxamide sidechain functionality in asparagine and glutamine residues. Typically, the correct assignment can be made based simply visually based on the H-bonding network commonly seen around these groups.

If you are using a pdb structure as an input for anything, please make sure you carefully analyze it yourself prior to use, visually and using standard computational tools. A lot of junk is just getting thrown into the database these days.

Permalink to Comment

12. Anonymous on August 16, 2013 12:05 PM writes...

@11: Indeed, C, N and O atoms are difficult to distinguish by X-ray crystallography, because they are similar in electron density, while H atoms never show up. So assigning these atoms must be done carefully (and checked) when there are more than two orientations in pseudo-symmetric amino acid side chains, like His, Asn and Gln, by looking at their interactions with neighbouring atoms.

However, my main point was that over-fitting of data is the biggest problem, and can only be avoided by looking at the correlation between observed results and a portion of data that has not been used in refinement. It seems this Cryo-EM group has overlooked this issue - a schoolboy error!

Permalink to Comment

13. Curious Wavefunction on August 16, 2013 12:21 PM writes...

Fitting data to cryo-EM seems to be a good example of what psychologist and skeptic Michael Shermer calls "patternicity", the tendency engineered into our primitive, Savannah-dwelling brains to ascribe signal to noise to avoid making false negative errors that could have weeded us out of the gene pool. It's like seeing the Virgin Mary in a piece of wood bark.

Permalink to Comment

14. Imaging guy on August 16, 2013 12:28 PM writes...

# 9.Casual Observer
This article shows how that is done.
"A method for the alignment of heterogeneous macromolecules from electron microscopy", Maxim Shatsky et al, Journal of Structural Biology 166 (2009) 67–78.

Permalink to Comment

15. Anonymous on August 16, 2013 12:46 PM writes...

@13: That's one of my big concerns with Big Data, and we see it all the time in post-hoc clinical data analysis, leading to hopes of success in Phase 3, based on patterns in Phase 2 data that is nothing more than noise.

Permalink to Comment

16. anon on August 16, 2013 12:54 PM writes...

a little bit dubious to say that protein crystallography is easier. crystallization is a pretty big hurdle.

Permalink to Comment

17. Anonymous on August 16, 2013 12:57 PM writes...

@17: Back in my PhD days they used to call me Buffer Boy, because I had to make up over 3,000 buffers to get my first decent crystals. :-)

Permalink to Comment

18. Derek Lowe on August 16, 2013 1:07 PM writes...

#9, I've put up a link to the Einstein-picture result. It's in figure 2 of the paper (open access).

#1, 11, 12: absolutely right. X-ray crystals are to be given the fishy eye as well, thus that "not the magic wand" link in the second paragraph. I think that's why cryo-EM is such a beast. Think about how many things can go wrong with, say, a 2.4 A x-ray structure. And a good cryo-EM structure is what, 9 to 12 angstroms? Everyone's going wild about this possible 6 A resolution, and for someone that's used to X-ray structures, that's like groping around in the dark with ski gloves on. I feel for everyone doing that work; they're right on the edge of what can be accomplished.

Permalink to Comment

19. Casual Observer on August 16, 2013 1:09 PM writes...

@14: Thanks very much, it's a terrific demonstration of how scarily powerful overfitting your data can be. (Applies to many other things as well as scientific data, I think.)

Permalink to Comment

20. Casual Observer on August 16, 2013 1:20 PM writes...

And @18, thanks Derek as well!

Permalink to Comment

21. NMH on August 16, 2013 1:49 PM writes...

@17: well, at least you got decent crystals. I never did, except one, that had a unit cell for a salt. And I have the publication record to show for it. That's why I got out of crystallography.

Beware of any PI who has a specific aim to purify a protein and get crystals, and wants to convince a grad student to do it.

Permalink to Comment

22. Anonymous on August 16, 2013 1:58 PM writes...

@21: It helped to be a masochist. :-)

At least I had the computing power and algorithms I needed to solve the sructure. My PI was Max Perutz, who had to use punch cards to store all the X-ray data (and a small army of attractive female assistants to manage them all).

Permalink to Comment

23. Anonymous on August 16, 2013 2:05 PM writes...

@22: I was a masochist. I purified my protein in a glove box in a cold room, in the middle of the night.

I have not been the same ever since. :(

Permalink to Comment

24. hn on August 16, 2013 4:11 PM writes...

Crystallization is not always a big hurdle. Some proteins are easy to crystallize. Yours may just be one of the lucky ones. Don't give up before you try.

Permalink to Comment

25. Sili on August 17, 2013 3:18 PM writes...

Michael Shermer calls "patternicity"
Why am I not surprised he felt the need to make up his own word for "pareidolia". Permalink to Comment

26. Secondaire on August 17, 2013 4:44 PM writes...

Speaking of dodgy crystal structures, there is a sub-group within my group who is collaborating to try and get a structure of an inactivated enzyme + inactivator complex , and the collaborators (who seem to know not much of anything about enzymes) keep doing things like trying to obtain structures without the cofactor present (d'oh!), or under conditions that are shown to NOT actually inactivate the enzyme, and then they keep wondering why what comes out the other end makes no sense.

Permalink to Comment

27. Anonymous on August 17, 2013 5:05 PM writes...

Here's a really good article on misinterpreting data:!

Permalink to Comment

28. cliffintokyo on August 18, 2013 6:33 AM writes...

If Cryo-EM provides some structural information about HIV surface proteins gp120 and gp41 that we do not/cannot get using another protein structure elucidating technique, then we need to have a very transparent and serious discussion about the validity of the work by the Sodroski group, and perhaps the validity of Cryo-EM technology -
because we want to use such important structural information for med chem design purposes.
If not, we do not need to get over-excited about the degree of resolution!
@#27: This seems to more about over-interpreting data, judging from the comments, although I am not familiar with this technology.

Permalink to Comment

29. Anonymous on August 18, 2013 7:58 AM writes...

@28: The article also explains why and how we tend to over-interpret data.

I posted it only because it seems relevant to this discussion...

Permalink to Comment

30. zmil on August 18, 2013 5:49 PM writes...

Sat in on a presentation of this work a little over a year ago. I have never, ever seen such utter disbelief of a presentation. They brought in a non-retrovirologist cryo-EM expert because they weren't sure anyone else had enough structural background to really critique the project, and he just kept saying, over and over, "you don't understand, this is *impossible!*" It was interesting at first but then it got so uncomfortable I was wishing it would stop.

Also, the structure he presented at that meeting? Wayyy more detailed than in this paper. He was pointing out individual residues and crap. Never seen anything like it -well I have, but only with X-ray crystallography. Wish I knew enough to make an informed judgement on the merits, but sadly all I can do is sit in the cheap seats munching popcorn...

Permalink to Comment

31. AGMMGA on August 19, 2013 2:51 AM writes...

@11: side chain issues and general geometry issues for the protein can be sorted out pretty easily by algorithms nowadays. If you don't wanna go through th pain of checking everything yourself, check out pdb_redo ( They re-refine old data with the latest algorithms, which spot these problems and automatically correct them. Also, molprobity ( might help to flag any problematic areas.

Permalink to Comment

32. Anonymous on August 19, 2013 4:49 AM writes...

Maybe we could get a 1-Angstrom molecular structure if we just merge a trillion brass rubbings?

Permalink to Comment

33. Spoons on August 21, 2013 10:24 AM writes...

This reminds me of my Biochemistry class. Since biochemistry was a sub major of chemistry our professor went to great lengths to present papers where some biologists tried to do a lot of hand waving and yadda-yadda when it came to the important underlying chemistry elements of a biology paper. Now I realize she was trying to get our BS-detectors trained up. Same thing with my instrumental analysis class.

Permalink to Comment

34. JTA on November 6, 2013 1:16 PM writes...

Pop corn time is back!
In this week's PNAS, three lengthy critiques are published (by Richard Henderson, Marin van Heel, and Sriram Subramaniam), as well as a lengthy reply from the authors. From my quick reading, the original authors confront the critiques only selectively. For example Henderson's Figure 2I-L are damning enough on their own but are not addressed in the author's reply.

Permalink to Comment


Remember Me?


Email this entry to:

Your email address:

Message (optional):

The Last Post
The GSK Layoffs Continue, By Proxy
The Move is Nigh
Another Alzheimer's IPO
Cutbacks at C&E News
Sanofi Pays to Get Back Into Oncology
An Irresponsible Statement About Curing Cancer
Oliver Sacks on Turning Back to Chemistry