A study published Wednesday in the journal PLOS ONE suggests that measures currently afoot in nearly every area of science to increase the transparency requirements for research will mean we can expect to see more of these seemingly dull results in the future -- and that's a good thing. Far from boring, those trials that find a drug doesn't do what we hoped can be equally as important -- or even more so -- than the ones that do.
Take the Women's Health Initiative, a giant, long-term study launched in 1991 that tested, among other things, the effects of hormone therapy in women who had gone through menopause. For decades, hormone replacement had been seen as a kind of fountain of youth by women and doctors. But the careful study found that estrogen alone increased risk of stroke. Estrogen combined with another hormone increased risk of breast cancer. It wasn't a positive result, but it changed medical practice, and a recent analysis predicts that the information on combined hormone therapy saved 126,000 women from dying of breast cancer between 2003 and 2012.
"If there’s anything disappointing about this, it's that greater transparency might result in discovering that there are fewer significant benefits from some interventions than we had thought," said Robert Kaplan, chief science officer of the Agency for Healthcare Research and Quality. "It is, in my opinion, a significant improvement in science."
Kaplan and Veronica Irvin from Oregon State University set out to study the effect transparency requirements that went into effect around the year 2000 had on clinical trial results. They expected they might see fewer positive results once researchers were required to report ahead of time what they were planning to study and how. And that's exactly what they saw.
They examined 55 large clinical trials funded by the National Heart, Lung, and Blood Institute between 1970 and 2012. Trials either found a benefit, a harm, or a null result -- meaning that the trial did not find evidence that the intervention affected the intended health outcome.
The year 2000 turned out to coincide with a clear shift, from a preponderance of positive, beneficial results, to a preponderance of null results. The graph below shows the details: more than half of 30 cardiovascular disease trials published before 2000 found that drugs or dietary supplements had a positive effect. Of the 25 studies published after the transparency standards were introduced, only 2 were positive -- less than 10 percent.
The researchers can't say whether the transparency requirement was the cause of the dramatic decrease in positive trials; there are many factors that could influence clinical trial findings.
But they did try and rule out other possible factors. For example, a common question about later trials is whether they are held to a higher bar because they're being compared to established treatments. The earlier treatments might have just had to demonstrate they were better than a placebo that was no treatment at all. But in the two data sets, nearly the same proportion of trials were compared to placebos.
The researchers also intentionally used only very large trials that received more than half a million dollars in funding each year. That's because those trials' results are likely to be published, regardless of what they show. That meant the research team was immune from a related problem in science and medicine generally, in which many results that show null results are presumed to never be published at all.