The FDA, in an analysis of subgroups from an observational study, had concluded that one of those subsets of patients had a substantial reduction in mortality as a result of being treated with plasma.
“Many of you know I was a cancer doctor before I became FDA commissioner,” Hahn said at the White House. “And a 35 percent improvement in survival is a pretty substantial clinical benefit. What that means is — and if the data continue to pan out — 100 people who are sick with covid-19, 35 would have been saved because of the administration of plasma.”
But the 35-out-of-100 claim wasn’t accurate, scientists said Monday. The FDA commissioner appeared to have mixed up absolute risk reduction and relative risk reduction, which are basic concepts in economics and in the presentation of data from clinical trials.
“I’m absolutely incredulous,” said Peter Lurie, a former top FDA official and now the president of the Center for Science in the Public Interest.
On Monday night, Hahn in a tweet acknowledged he had misspoken during the news briefing about the findings of the convalescent plasma study.
“I have been criticized for remarks I made Sunday night about the benefits of convalescent plasma. The criticism is entirely justified,” Hahn wrote. “What I should have said better is that the data show a relative risk reduction not an absolute risk reduction.”
David Steensma, an oncologist at the Dana-Farber Cancer Institute in Boston and a faculty member at Harvard Medical School, tweeted earlier that “The head of the United States Food and Drug Administration (!) @SteveFDA does not understand the difference between relative and absolute risk reduction — a basic, fundamental concept in medicine, statistics and clinical trial interpretation. It just gets worse and worse.”
Essentially, the Trump administration figures had compared one group of patients who got a certain kind of plasma with a group who got a different concentration at a different point in the disease, thus showing the relative difference between those groups. It was not a measure of what happens when some patients get plasma and some don’t — the kind of research necessary to send a signal of whether a treatment is truly helping. The FDA also considered data from other studies.
Robert Califf, who was FDA commissioner during the Obama administration, called on Hahn on Twitter to correct his statement. “It would be good for Steve to publish a correction,” he said. “I’m sure he meant to correctly state the translation from relative effect to absolute effect.”
Eric Topol, director of the Scripps Research Translational Institute, said on Twitter that data gathered on the use of plasma in covid-19 patients indicates that 3 people out of 100 would be saved at seven days of treatment and 5 at 30 days — not 35.
The FDA declined to answer questions Monday about Hahn’s statement. But the controversy comes at a particularly fraught moment. In recent weeks, Hahn has been trying to convince the public that the FDA will make upcoming coronavirus vaccine decisions based only on data and science. Scientists say it is a dangerous time for the FDA to take a hit to its credibility.
Remarks by Trump and Health and Human Services Secretary Alex Azar on Sunday also raised eyebrows among scientists — or drew ire.
Trump called plasma a “historic breakthrough,” which scientists insist it is not. And he and Azar repeatedly highlighted the 35 percent reduction in mortality. Even FDA scientists said more studies need to be conducted to get definitive evidence about plasma’s efficacy.
But scientists said the study was not the kind that could draw that type of conclusion. It was not a gold standard trial typically used to make decisions about whether a drug is safe and effective. In those studies — known as randomized controlled trials — certain patients get the real medication that is being analyzed while others receive a placebo. That way, scientists can establish with greater certainty that a medication works and that it is not causing dangerous side effects.
The data on plasma came from a nationwide program run by the Mayo Clinic that gave hospitalized patients early access to convalescent plasma, which is obtained from people who have survived covid-19 and who have marshaled an army of disease-fighting antibodies. The FDA oversaw the program, which has treated more than 70,000 patients. It was an observational study, which is used to generate a hypothesis. FDA officials acknowledged that randomized clinical trials were needed to provide definitive answers about plasma.
“We saw about a 35 percent better survival in the patients who benefited most from the treatment, which were patients under 80 who were not on artificial respiration,” Azar said. “I just want to emphasize this point, because I don’t want you to gloss over this number. We dream in drug development of something like a 35 percent mortality reduction. This is a major advance in the treatment of patients. This is a major advance.”
To people not familiar with the underlying data, that might have made it seem like people receiving plasma from covid-19 survivors were 35 percent more likely to survive than those who did not receive plasma. But the studies aren’t designed to show that. And the data appeared to apply to a subset of patients: those under age 80 who weren’t tethered to breathing machines.
Outside scientists said the administration officials who touted a 35 percent reduction in death risk were overstating the conclusions that could be drawn from suggestive studies without comparison groups. They said that because plasma has not been studied in randomized, controlled clinical trials, it is still too early to draw conclusions about its efficacy or even when, how and in what quantities doctors should administer it.
“Right now, we don’t have any randomized controlled data to suggest that convalescent plasma is useful for covid-19. They took the Mayo Clinic data last night and were talking about a 35 percent reduction in mortality. There’s a big difference in absolute and relative risk in describing clinical trial results,” said Gregg Gonsalves, an assistant professor at the Yale School of Public Health. “It’s very hard to believe the head of the FDA and the head of Health and Human Services does not know that distinction.”
The FDA, in a statement issued Monday, said the agency had reviewed data from “multiple sources, including over a dozen published studies, as well as the prepublication data from the Mayo Clinic.” The agency noted that “there appears to be roughly a 35 percent relative improvement in the survival rates of patients who were treated with higher-titer plasma when compared to those treated with lower titer plasma.” A higher-titer plasma has a greater concentration of antibodies.
That referred to the subset of patients. In an FDA memo justifying the use of plasma, one bullet point concludes, “There was no difference in 7-day survival in the overall population between subjects transfused with high versus low titer” convalescent plasma.
The White House issued a statement defending the decision to issue an emergency use authorization for convalescent plasma.
“Convalescent plasma is a proven therapeutic method, and if it saves one life then it is worth having it on the market and available,” the White House statement said. “President Trump believes all Americans should have access to proven, safe, and affordable treatment options.”