The NBER (National Bureau of Economic Research) has been looking at the patterns of scientific publication and grant awards in the US, and has noticed some interesting trends. According to Inside Higher Ed, the study found (first off) that scientific publications are increasing at about 5.5% a year, and the report suggests that this might mean that any individual who reads at the same rate is seeing their own current knowledge decrease by the same amount.
I'm not so sure about that. While there are indeed more papers every year, the marginal utility of each new paper isn't necessarily very high - if I can switch into econ-speak myself. That's especially true if increased numbers of articles are due to new journals that end up (directly or indirectly) pulling things into the literature that wouldn't have even been published otherwise, simply because journals need to fill their pages. That said, the volume of interesting science done (and to be read about) each year is still increasing - I certainly can't deny that - but it would be a mistake to assume that "Scientific Journal Publications" are some sort of homogeneous good that can be measured as such.
Two other trends that were spotted make more sense to me: one is that the average number of co-authors is rising steadily. You wonder if that last part is just due to those physics papers that have six hundred people on them, but it seems to be the case across all disciplines. There are fewer and fewer solo scientific publications than there used to be, which confirms my own experience looking across the the chemistry literature.
Another trend is that fewer highly-cited big-news papers are coming from the younger end of the age distribution. The report says that "Peak productivity has increased by about 8 years, with the effect coming entirely from a collapse in productivity at young ages." The average ages for discoveries that later went on to win Nobels has been going up, as has the average age at which a scientist appears on their first patent. And that's worth thinking about - is it that our educational setup in the sciences sends people out into the fray at later and later ages? Or that the disciplines themselves have gotten more complicated, requiring a longer period before a substantial contribution can be made?
I think that a big factor is that younger scientists probably feel insecure working on high-risk high-reward projects. In academia, they're fighting for grant money and tenure, and I think that many people in that situation are careful about balancing "exciting and groundbreaking" against "likely to produce solid, publishable results". And industrial scientists tend to need more experience before they can make a big discovery as well, since the more applied fields have a larger body of specific knowledge built up.
The report contrasts these trends against the long-held image of the brave young researcher pushing toward a big discovery. I'd argue that the Nobel itself suffers from this problem, with its strict three-names-only rule. It's my impression that the committees that decide the prize have been having a harder and harder time of it over the years trying to find a way to stick to that. It has (inevitably) led to a number of deserving people getting left out - as well as a number of deserving discoveries that couldn't be narrowed down well enough. (Organic chemistry has the metal-catalyzed couplings as an example).
Finding ways to recognize large (often interdisciplinary) teams would be one step. Another change that might need to be made could include easing up a bit on the younger grant recipients, realizing that it's going to be increasingly difficult for them to hit things out of the park at that point in their careers. Could that also allow some of the better ones to work in tougher areas, with less fear of the consequences of failure?