Here's an article in Angewandte Chemie that could probably have been published in several other places, since it's not specifically about chemistry. It's titled "The Organization of Innovation - The History of an Obsession", from Caspar Hirschi at St. Gallen in Switzerland, and it's a look at how both industrial and academic research have been structured over the years.
He starts off with an article fromThe Economist on the apparent slowdown in innovation. This idea has attained wider currency in recent years (Tyler Cowen's The Great Stagnation is an excellent place to start, although it's not just about innovation). I should note that the Economist article does not buy into this theory. Hirschi dissents, too, but from another direction:
Despite what the authors would have us believe, the “innovation blues” lamented in the The Economist have little to do with the current course of technological development. The source of the perceived problem arises instead from a sense of disappointment over the fact that their innovation theory does not hold up to its empirical promise. The theory is made up of a chain of causation that sees science as the most important driving force behind innovation, and innovation as the most important driving force for the economy, and an organizational principle maintaining that the three links in the chain function most efficiently under market-oriented com- petition.
The problem with this theory is that only the first part of its causal chain stands up to empirical scrutiny. There is every indication that progress in scientific knowledge leads to technical innovations, but it is highly unlikely that a higher level of innovation results in greater prosperity. . .
He then goes back to look at the period from 1920 to 1960, which is held up by many who write about this subject as a much more fruitful time for innovation. Here's the main theme:
A comparison between these previously used formulas with those used today strongly suggests that our greatest stumbling block to innovation is our theory-based obsession with innovation itself. This obsession has made scientists and technical experts play by rules stipulating that they can deliver outstanding results only if they are exposed to the competitive forces of a market; and if no such market exists— as in the case of government research funding—a market has to be simulated. Before 1960, the organization of innovation had hewed to the diametrically opposite principle. . .
I have a couple of problems with this analysis. For one, I think that parts of the 19th century were also wildly productive of innovation, and that era was famously market-driven. Another difficulty is that when you're looking at 1920-1960, there's the little matter of World War II right in the middle of it. The war vastly accelerated technological progress in numerous areas (aeronautics, rocketry, computer hardware and software, information science and cryptography, radar and other microwave applications, atomic physics, and many more). The conditions were very unusual: people were, in many cases, given piles of money and other resources, with the understanding that the continued existence of their countries and their own lives could very well be at stake. No HR department could come up with a motivational plan like that - at least, I hope not.
Hirschi surveys the period, though, with less emphasis on all this, but it does come up in his discussion of Kenneth Mees of Eastman Kodak, who was a very influential thinker on industrial research. It was his theory of how it should be run that led to insitutions like Bell Labs:
The key to success of an industrial laboratory, he explained, lay in the ability of its directors to recreate the organizational advantages of a university in a commercial setting. Industrial scientists ought to be given the greatest possible latitude to conduct their research as they see fit, with less outside interference, flat hierarchies within the institution, and department heads who are themselves scientists. Like professors at universities, he contended, the senior scientific staff should hold permanent appointments, and all scientists ought to have the opportunity to publish their research results.
Readers here will be reminded of the old "Central Research" departments of companies like DuPont, Bayer, Ciba and others. These were set up very much along these lines, to do "blue sky" work that might lead to practical applications down the road. It's absolutely true that the past thirty years has seen most of this sort of thing disappear from the world, and it's very tempting to assign any technological slowdown to that very change. But you always have to look out for the post hoc ergo propter hoc fallacy: it's also possible that a slowdown already under way led to the cutbacks in less-directed research. Here's more on Mees:
For Mees, industrial research was a “gamble”, and could not be conducted according to the rules of “efficiency engineering”. Research, he insisted, requires a great abundance of staff members, ideas, money, and time. Anyone who is unwilling to wait ten years or more for the first results to emerge has no business setting up a laboratory in the first place. Mees established the following rule for the organization of scientific work: “The kinds of research which can be best planned are found to be those which are least fundamental.” But because Mees regarded the basic sciences as the most important source of innovation, he advised research directors to try not to rein in their scientists with assignments, but instead to inspire them with questions.
I don't find a lot to argue with in that sort of thinking, but that might be because I like it (which doesn't necessarily mean that it's true). I hope it is, and I would rather live in a world where it is, but things don't have to be that way. I do think, though, very strongly, that the application of what's called "efficiency engineering" to R&D is a recipe for disaster. (See here, here, here, here, and here for more). And there are people in high places who apparently agree.
The Ang. Chem. article goes on to note, correctly, that many of these big industrial research operations were funded by monopoly (or near monopoly) profits. AT&T, IBM, Eastman Kodak and others began to use their research arms as public relations tools to argue for that status quo to continue.
Because monopolies could not be justified directly, the only route the companies in question had open to them was a detour that required more and more elaborate displays of their capacity for innovation. Once again, architecture proved to be well suited for this purpose. In the late 1950s and early 1960s, several American industrial groups built new research centers. They opted to locate them in isolated surroundings in the style of a modern university campus, and favored a new architectural style that moved away from the traditional laboratory complexes based on classic industrial buildings such as the one in Murray Hill. . .
The architecture critics of the time soon came up with an apt name for these buildings: “Industrial Versailles”. The term was fitting because the new research centers were to industrial innovation what Versailles had been to the Sun King: complexes of representation, as the historians of technology Scott Knowles and Stuart Leslie have detailed.
We actually owe a lot of our current ideas about research building design to the thoughts about what made Bell Labs so productive in the 1950s - as you keep digging, you keep finding the same roots. Even if those theories were correct, whether the later, showier buildings were true to them is open for debate.
Hirshci finishes up his piece with the 1957 Sputnik launch, which famously had a huge effect on academic science funding in the US. I only realized when I was in my 20s that my whole impression of the science facilities in my own middle and high school in Arkansas were shaped by that event. I'd sort of assumed that things like this were just always ten or twenty years old, but that was because I was seeing the aftereffects of that wave of funding, which reached all the way to the Mississippi Delta. Here's Hirschi on the effects in higher education and beyond:
The explosion of government research funding resulted in serious quandaries about how best to allocate these funds. There were countless research institutes, and there was a need for clear rationales as to which institutions and individuals would be entitled to how many dollars that came from taxes. An attempt was made to meet this challenge by introducing an element of marketlike competition. Artificial competition for project-related subsidies, to be regulated and controlled by the funding agencies, was set in motion. Successful proposals needed to provide precise details about the scope of each project and a set time frame was assigned for the completion of a given project, which made it necessary for grant seekers to package fundamental research as though it were application-oriented. This set-up ushered in a period in which innovations were proclaimed well before the fact, and talked up as monumental breakthroughs in the quest to secure funding. Representation became integral to production.
It did not take long for this new regime to have drastic reverberations for industrial research. The flood of money that inundated the research universities heightened the incentive for industrial groups to outsource costly laboratory work to universities or public research centers. At the same time, they were inspired by the public administration's belief in the rules of the market to pay heed in their own research divisions to the credo that the innovative impulse requires the intensity of a competitive situation. In the long run, the private sector did its part in making the new form of market-oriented project research the only accepted organizational principle.
To my eye, though, this whole article wraps up rather quickly like this. It seems like a reasonable short history of research organization in the mid-20th century, followed by several assertions. Hirschi's not advocating a return to the 1950s (he explicitly states this), but it's hard to say what he is advocating, other than somehow getting rid of some of what he seems to feel is unseemly competition and market-driven stuff. "The solution can only lie in the future" is a line from the last paragraph, and I hope it reads better in German.