A third of the clinical trial results that federal regulators reviewed to approve drugs made by large pharmaceutical companies in 2012 were never publicly reported, according to a new study that grades companies on their transparency.
To assemble the report, a handful of dogged researchers pored over thousands of pages of regulatory documents, counting up the number of trials Food and Drug Administration regulators reviewed, versus how many trials were published or publicly reported. The result is an inaugural transparency report card -- essentially an index showing just how much of the evidence about how the 15 drugs actually work in people is publicly available.
"Right now, there's a big push for evidence-based medicine. It’s impossible to have evidence-based medicine without the evidence," said Jennifer Miller, an assistant professor in the division of medical ethics at New York University School of Medicine, who oversaw the report. "If the public evidence is partial or biased you risk having partial or biased [prescription] guidelines," on how to give the drugs to patients.
The study, published Thursday in the journal BMJ Open, shows gaps in transparency, but has limitations. It analyzed only a subset of drugs approved in a single year, 2012. The researchers also only examined whether clinical trials were registered and reported, not what that data suggested about how the drugs worked.
Clinical trials are normally conducted in sequence, with the earliest stage, phase I trials, designed to determine a drug's safe dose. Phase III trials, the largest and most authoritative trials, are designed to test the safety and effectiveness of drugs. According to 2007 FDA requirements, later-stage trials meeting certain requirements are generally required to be reported and registered. Only a small fraction of the trials the study identified were required to be disclosed for each drug, but five drugs made by three companies were rated as zero percent compliant. Three of the 15 drugs studied had at least one publicly unavailable late stage phase III trial.
An FDA spokesman did not immediately respond to a request for comment.
Miller has bigger plans going forward -- her non-profit, Bioethics International, which is funded by foundations and universities but not pharmaceutical companies -- will publicly grade the transparency of drug companies year by year, in a report card format.
The idea of the "Good Pharma Scorecard" is that the public disclosure will provide an added incentive for companies to be more transparent. The researchers gave the companies each a transparency grade, based on what percentage of all trial results -- not just the ones required to be reported -- were made publicly available.
Among the results:
- Gilead's HIV drug Stribild got a grade of 21 percent. Of 34 trials presented to the FDA, 26 were never made public.
- Bayer's Stivarga, a colorectal cancer drug, got a grade of 42 percent. Of 12 trials presented to the FDA, 7 were never made public.
- Pfizer's kidney cancer drug, Inlyta, got a grade of 100 percent. Results of all 28 of the trials presented to the FDA were made public.
"To me the right number is 100 percent; if you’re selling a product on the U.S. market, the public and the clinical science community has a right to be able to review the results, and has a right to know on what basis" the drug is approved, said Joseph Ross, an associate professor of medicine at Yale University School of Medicine, another of the authors of the paper.
Three companies were in compliance with FDA required disclosures and also scored well on transparency: Pfizer, GlaxoSmithKline, and Johnson & Johnson.
To understand why a selective look at the evidence might matter, given that regulators with full access to the evidence already determined a drug is safe and effective, Ross pointed to Zaltrap, a colorectal cancer drug that was approved in 2012.
According to the study, the FDA reviewed 30 trials when considering the drug made by Sanofi, but only 12 of those trials were ever published or publicly reported, for a transparency score of 40 percent. That means that physicians trying to decide how to use it had only a limited look at the evidence of how it worked in patients, Ross said -- potentially critical information since the drug initially cost more than $10,000 a month and there was an existing medicine on the market. The study did not delve into what the nonpublished trials showed in each case, so it could not determine whether they would have provided usable information.
"We look forward to reviewing the study to understand the authors’ methodology and results. Sanofi is committed to the timely, effective sharing of clinical trial data in an effort to help patients, healthcare professionals, patient advocacy organizations and the research community," Sanofi spokeswoman Mary Kathryn Steel wrote in an e-mail. "Sanofi registers its clinical trials on public registries ... and results of clinical trials included in these registries are published on Sanofi.com, in peer-reviewed medical journals and at medical congresses."
For years, it's been known that many study results often go unpublished, languishing in file drawers, with concerns that those studies tend to be the ones with ho-hum or negative results. While some kinds of studies carry more weight than others, many researchers argue that seeing all the evidence can be informative, perhaps suggesting that a drug works better for one group of patients than another. And Miller argues that selective publication can also erode the trust that physicians and patients have in the drug and the company.
Bayer, another company that scored poorly on transparency and on its adherence to FDA guidelines -- with a zero percent compliance for its colorectal cancer drug Stivarga on the subset of trials required to be reported -- said that it supports transparency. In addition to posting trials on its own website, Bayer shares information through an independent website which allows researchers to request clinical data for new medicines approved after Jan. 1, 2014, spokeswoman Rose Talarico said.
"We will continue to work with internal and external experts and stakeholders to continually evolve best practices for creating greater transparency to advance medical knowledge, public health and science," Talarico said.
Other cases have shown that selective bias in which studies are published may provide a limited picture of how drugs work (or don't).
A 2008 study in the New England Journal of Medicine examined evidence provided to the FDA in support of a dozen antidepressants. Nearly two dozen studies that showed the drugs didn't work were not published. Eleven others were "published in a way that, in our opinion, conveyed a positive outcome," the authors wrote. Only 3 studies showing negative results were published. Overall, nearly a third of the studies submitted to the FDA weren't published.