As Pfizer eyed a £70bn takeover of AstraZeneca this spring, the prospective deal highlighted an awkward truth for the chief executives of both pharmaceutical groups: neither had a healthy stock of experimental drugs in their own pipelines.

While recent months witnessed a “biotech bubble”, with surging valuations for fledgling drug developers on public markets – notably in the US – the broader picture of innovation for new medicines remains disappointing.

Overall, there has been a slight increase in the number of products approved by US and European regulators in the past few years, but the underlying trend is broadly unchanged.

That may partly be because tougher standards have increased the rate of refusals; but the number of applications for drug authorisations has remained all but flat, despite the billions of dollars spent on research.

For a long time, larger drug companies have sought to boost productivity by bringing greater scale to discovery. That has allowed them to invest in costly techniques such as high-throughput screening of new compounds. Many have also sought to buy companies with better pipelines to compensate for expiring patents on the products in their own portfolios. Yet investors have been pushing back against companies’ attempts to spend their way out of trouble and continue to invest in “bricks and mortar”.

“How do you combine innovation and rapid product development when bringing together two of the largest pharma entities on the globe?” asks Steve Brozak, president of WBB Securities. “It’s an oxymoron, like ‘military intelligence’.”

Jo Walton, pharmaceutical analyst with Credit Suisse, also highlights the poor returns on all the money spent by industry in recent years. When she plots R&D spending against estimated peak sales for drugs that will be launched in coming years, Pfizer ranks among the lowest in the industry. AstraZeneca is not far ahead.

A recent study by SSR, a US-based research firm, shows a similar pattern. The message? Small is beautiful.

Mid-sized quoted biotech companies have generally provided far better returns than their larger peers. Of course, many even smaller – and unquoted – biotech companies fail too.

But smaller scale is an approach that the larger pharmaceutical companies have all sought to emulate as they seek to improve their rates of innovation. Most have restructured round more modest biotech-sized units, giving greater freedom and accountability to scientists.

Francesco de Rubertis, a partner with Index Ventures, a venture capital company, says: “One of the key differentiators between winning companies and lagging ones is the nature of people in decision-making positions. The empowerment of people close to the action, enabling decisions to be taken by those with their ears to the ground, is better.”

Bernard Munos, a consultant to industry, praises larger companies such as GlaxoSmithKline and Johnson & Johnson for just such a shift to a more entrepreneurial, small-scale culture, which he says is beginning to pay off with higher numbers of new drug filings.

“Some have been successful at producing more and better innovation,” he says. “But the industry has not addressed the third requirement to secure the future of drug development: affordability. You need to take a major chunk of cost out of the system.”

He argues that as healthcare systems resist the rising costs of new drugs, further disruption – probably driven by companies outside the pharmaceutical industry – is required.

He points in particular to the role of data companies and the use of biosensors to collect far more information on patients as essential factors to improved understanding and more efficient drug development.