The study, published in the Journal of Clinical Investigation, drew concern from specialists in clinical testing. Although some tests were in good agreement across all three providers, such as triglycerides and red blood cell counts, others diverged -- "good" HDL cholesterol and white blood cell counts, for example. Theranos was 1.6 times as likely to report abnormal results as Quest Diagnostics and Laboratory Corporation of America Holdings.
Theranos is a Silicon Valley darling, a startup once valued at about $10 billion, that sought to disrupt the big business of blood testing by offering rapid, cheap, finger stick blood tests. The company pursued a radical strategy: it posted its price list online and supported an Arizona law that allowed consumers to order any blood test without a prescription. This democratization of health information could be tremendously empowering, however, experts worried that if Theranos' results are erroneously flagged as abnormal or are simply low as the new study suggests, it could lead consumers astray.
"People know if your cholesterol is over 200, perhaps you need to do something about that. And they know that because a '200' is a '200,' pretty much anywhere you get your testing done," said Patricia Jones, the president of the American Association for Clinical Chemistry. "The fact that Theranos's results are significantly different, I find concerning. There is allowable error in diagnostics, but according to the paper, Theranos exceeded the total allowable error in that test."
Theranos's laboratory director Daniel Young, however, said the study suffered from "sloppiness" and pointed out that it simply shows discrepancies between testing services and doesn't compare the measurements to ground truth. It's possible, he said, that Theranos's results are the more accurate, correct ones.
"What they’re trying to do is suggest that Theranos's results are less accurate or less reliable, but this study doesn’t support that at all," Young said.
The study began in July 2015, before the hype about Theranos was punctured by a Wall Street Journal investigation that raised deep questions about the company's secretive technology. Months before those revelations, a team at the Icahn School of Medicine at Mount Sinai was curious about Theranos, a company that had generated much media coverage, while largely staying mum on the details of how its technique worked -- generating skepticism in the scientific community.
"We definitely see a future where you’re doing more regular blood testing to assess wellness and supporting precision medicine, and these new areas of research, so we’re not coming at it from the perspective that we’re suspicious about Theranos," said Joel Dudley, director of biomedical informatics at Mount Sinai. "In fact, our main motivation was it would be great to use the technology because low-volume blood draws would be an enabler of more regular testing."
What they found was some variability across testing services generally, but most notably with Theranos. The researchers asked 60 healthy adults to give finger stick blood samples at Theranos Wellness Centers in Arizona and compared those results with blood drawn from a vein and tested by lab company behemoths Quest and LabCorp. Each person had 22 measurements taken by each service, and two-thirds of them varied significantly between the different services.
A margin of variability is expected between testing services, and most of the discrepancies Dudley and his team found "wouldn't throw up huge red flags," he said. But Theranos denounced the study and wrote a critical letter to the journal pointing out problems, and Young said he felt publication should be delayed.
For example, Young said that all of the samples were collected after blood had been already drawn from a vein, which might explain the discrepancies between the services.
But Dudley responded by explaining that Young's critique was factually incorrect: one set of the venous blood draws had preceded the finger pricks and one set had come after. Dudley added that if Theranos' results can be thrown off by such factors, it should be disclosed.
The other two companies whose services were used in the study responded to the study less critically than Theranos.
"When you cut through all the data, the study points out a concerning lack of agreement between the reference labs [Quest and LabCorp] and the [Theranos] ‘low-volume-specimen’ lab, especially for heart health, where discrepant results can result in unneeded use of healthcare services, added expense, inconvenience for patients and doctors, or all three," Quest Diagnostics spokesman Dennis Moynihan wrote in an email.
LabCorp spokeswoman Pattie A. Kushner wrote that the quality of the specimen collection may matter in testing, and the company could not control for some of the factors that might affect the samples since they didn't collect them.
"Even without being able to fully account for these factors, the study shows that the results of the tests performed by LabCorp are well within established guidelines and meet performance expectations," Kushner wrote.
Alan Wu, the chief of the clinical chemistry laboratory at San Francisco General Hospital, who was not involved in the study, said that the results aren't surprising. Last year, he sent some medical students to use Theranos's technology and compare it with lab results, finding discrepancies -- especially among potassium levels. He noted that a fingerstick blood draw, a key part of the early marketing of Theranos's service, is notoriously difficult to do well. It can throw off the results if other fluids enter the sample as the finger is squeezed, or if there's muscle injury at the site of the finger stick.
"It's not something people would want to do if they knew the inaccuracies," Wu said. "The bottom line is I’m not surprised that we see differences; my understanding that the differences between Quest and Lab Corp. were minimal and the differences between those two and Theranos were where the majority occurred."
Theranos currently offers its proprietary finger stick technology for a single test, herpes, which was not included in the sample examined in the study.
Young argued there were clear mistakes: By looking internally at their own data, Theranos identified the people who had participated in the study and discovered that the authors had incorrectly classified some of the "good" HDL cholesterol levels as low when they were not in fact low according to Theranos criteria.
Dudley, who led the study, disclosed that he owns equity in a company called NuMedii that is focused on using data analysis to identify new uses for existing drugs. Another co-author of the paper, Eric Schadt, is on the scientific advisory board of NuMedii, but his connection was not disclosed in the paper. Schadt does not hold equity in the company and has not interacted with it for more than two years, so it did not trigger the journal's requirements for disclosing conflicts of interest, Dudley said.
Theranos spokeswoman Brooke Buchanan said that the company planned to publish results showing the validity of its tests sometime this year.
"The one thing that made laboratory professionals wary about it [Theranos] was because when we develop a new method, we publish it, we get it out there. And other people start using it and pretty soon everybody's using the method and it’s a good method," Jones said. "The fact that nobody knows anything about this method makes us wary."