Talking about pattern recognition leads a medicinal chemist to thoughts of SAR, structure-activity relationships. We spend a lot of time putting together tables of data - changes on one part of the molecule tabulated on one axis, changes in some other region on the other axis, and boxes filled in with the assay results.
And we spend a lot of time looking at all these piles of data and trying to see if they're telling us anything at all. Most of the time, they are - just not as much as we'd like. (If the project stubbornly refuses to make sense, though, we eventually find a reason to kill it. There's no point in working on something when you have no idea where you're going, why you are where you are, or how to get anywhere.)
No, most of the time what the data tell us are things like: "You can't put anything bulky here. Every time you do, the activity of the compound goes completely into the tank." Or "You can't put a basic nitrogen here - see that one-hundred-thousand-fold loss of activity?" We get mostly negative data, telling us about the things that don't work. Not that that's not valuable. It tells us to go find something else to do to the molecule, and not to waste much more time on that area.
The problem is, you can go for quite a long time without hearing anything positive, like "See? You had a methyl group here and you changed it to propyl. It helps! Maybe you should try butyl, and don't forget to go back in fill in that gap by making the ethyl." And even when you do get some data like that, there's always a limit to how much you can run with it.
I don't know if I've mentioned this before, but I'm convinced that there are no linear relationships in the real world. Everything will fall off the line if you push it hard enough in one direction or another. SAR trends tend to not stand up to much rough treatment at all: methyl, OK. Ethyl, a bit better. Propyl, great! Butyl. . .lousy. Pentyl (just to be sure,) terrible. Isopropyl (to see who's fooling who,) unspeakable. And so it goes. So much for the robust trend. You'll also get many parts of a molecule where just about anything fits OK. Nothing helps, but nothing hurts - until you finally reach the end of that empty cavity the stuff must be poking out into, and wham, back into the dumper.
What keeps you coming back are the occasional jackpots. Once in a while, you make something that (according to the model you've built in your head) should be another plain-vanilla compound, and for some reason it works. Time to change the model! There's always the real possibility that the next compound is going to break the mold, and most of the time the mold sorely needs breaking. So you keep at it.
The frequency with which those winners show up is similar to that used by slot-machine designers, to keep the customers pumping money in. (Quite a business model, that.) The more I think about it, the more I think that it's something the "Intelligent Design" people should look at. I'm not sure that this is the sort of Deity that they're thinking of, though. It's clear that Whoever's in charge of drug research has a really foul temper and a buttered-stairs sense of humor.