How do you know if someone's a good chemist? For now, I'll restrict that to "good bench chemist", because that's hard enough to answer on its own. People can put up a surprisingly good front, but there are some things to watch out for.
Knowing things like name reactions is just as often a blind as a good indicator. Those are a charmingly antiquated part of organic chemistry where reactions are referred to by their original discoverers (or popularizers, in some cases.) So you have ancient classics like the Dieckmann condensation or the Williamson ether synthesis, both of which are still in use every day of the week despite their 19th-century pedigrees, and more modern ones like the Suzuki coupling or the Heck reaction. There are scores of these things - one of these days I'll go on about them some more - and they're a semi-beloved, semi-loathed feature of all introductory organic classes. It's nice to have a good familiarity with them, because many of them are quite important, but name-dropping can also be a noisy distraction that tyros use to hide their other deficiencies.
Being quick with ideas and mechanisms up on the blackboard is usually a good sign, but not being so isn't always a bad one. Some people think differently than others, and at different paces. There are always people who need to go look out the window for a few minutes before coming up with the answer. The biggest quick-draw artists I've seen with blackboard reaction pathways have clustered at the top and bottom of my personal rankings. They were either very good chemists indeed, or that was about the only thing that they were good for.
Productivity at the bench is harder to fake, but it can be done in some cases. In the early days of combinatorial chemistry, some folks would hit on an easy-to-extend series of analogs (often by working out some handy way to set up and purify the reactions faster than usual), and run these out to impressive lengths to wow the onlookers. Back in the early 1990s you could really impress folks by suddenly turning in, say, 112 sulfonamides all at the same time, but now it's become a bit more commonplace. But being able to produce that many compounds means that you're at least fairly hard-working and organized, traits not to be underestimated.
One thing to watch is whether a person's chemistry can be duplicated by anyone else, and how good they are at duplicating things in turn. If someone consistently gets lower yields or messier products than other people running the same kind of thing, it's probably a warning sign. And if no one else can get a person's reactions to work as well as they did originally, it's almost always a red flag. It's very rare that someone has such consistently good hands that they always get higher yields. More likely, they're pulling the wool over your eyes, or over their own as well.
If you have a very green chemist coming into a lab - say, a summer undergrad or a first-year graduate student in academia - there's a foolproof way to test their bench skills. Have them reproduce a preparation from Organic Syntheses. That's a series (over 80 volumes worth) of useful procedures in organic chemistry. They're either preparations of particularly useful intermediates or illustrate new reactions and the preferred ways to run them. They're more detailed than the standard writeup in a chemical journal or patent (especially some patents I've seen), and they're checked by another research group entirely, with their comments appended. They're completely foolproof, and if you find someone who can't get one to work, odds are that you are, in fact, dealing with a fool.