The Washington PostDemocracy Dies in Darkness

Why we shouldn’t take peer review as the ‘gold standard’

It’s too easy for bad actors to exploit the process and mislead the public

White mouse in laboratory. (iStock)

In July, India’s government dismissed a research paper finding that the country’s economic growth had been overestimated, saying the paper had not been “peer reviewed.” At a conference for plastics engineers, an economist from an industry group dismissed environmental concerns about plastics by claiming that some of the underlying research was “not peer reviewed.” And the Trump administration — not exactly known for its fealty to science — attempted to reject a climate change report by stating, incorrectly, that it lacked peer review.

Researchers commonly refer to peer review as the “gold standard,” which makes it seem as if a peer-reviewed paper — one sent by journal editors to experts in the field who assess and critique it before publication — must be legitimate, and one that’s not reviewed must be untrustworthy. But peer review, a practice dating to the 17th century, is neither golden nor standardized. Studies have shown that journal editors prefer reviewers of the same gender, that women are underrepresented in the peer review process, and that reviewers tend to be influenced by demographic factors like the author’s gender or institutional affiliation. Shoddy work often makes it past peer reviewers, while excellent research has been shot down. Peer reviewers often fail to detect bad research, conflicts of interest and corporate ghostwriting.

Meanwhile, bad actors exploit the process for professional or financial gain, leveraging peer review to mislead decision-makers. For instance, the National Football League used the words “peer review” to fend off criticism of studies by the Mild Traumatic Brain Injury Committee, a task force the league founded in 1994, which found little long-term harm from sport-induced brain injuries in players. But the New York Times later discovered that the scientists involved had omitted more than 100 diagnosed concussions from their studies. What’s more, the NFL’s claim that the research had been rigorously vetted ignored that the process was incredibly contentious: Some reviewers were adamant that the papers should not have been published at all.

How gendered language leads scientists astray

Similarly, fossil fuel industry interests have tried to distort the public debate on climate change by sponsoring research and exploiting the prestige of peer review, undermining the overwhelming scientific consensus on the topic. For many years, climate skeptics published studies in the journal Energy & Environment, which had little credibility among climate scientists, even though it had a peer review process. (One of its articles said inaccurately that the sun is made of iron.) Typically, in the peer review process, editors choose reviewers, and E&E’s longtime editor, Sonja Boehmer-Christiansen, is an avowed climate change skeptic: “I’m not ashamed to say that I deliberately encourage the publication of papers that are skeptical of climate change,” she told the Guardian in 2011. Indeed, in the early 2000s, when scientists were just figuring out how much humans were changing the climate, public figures such as Sen. George V. Voinovich (R-Ohio) began citing research published in the journal in the name of delaying political action.

A few years ago, it emerged that Willie Soon, a scientist at the Harvard-Smithsonian Center for Astrophysics, had accepted $1.2 million from fossil fuel interests to publish studies, which he described as “deliverables,” in academic journals. (Much of his research has argued that variations in the sun’s energy can explain most recent global warming and that humans have had little effect on climate change, a thesis rejected by the majority of experts.) Peer review did not uncover these vested relationships: The editor of the Journal of Atmospheric and Solar-Terrestrial Physics told a reporter that it relied on authors to be truthful about conflicts of interest.

How blind reverence for science obscures real problems

Just as peer review fails to detect funding sources, it often also fails to catch ghostwriting, in which companies draft articles for academics who go on to publish them under their own names, sometimes with few or no changes. The editor of one journal told the Senate Finance Committee in 2010 that he suspected that at least a third of the papers submitted to his journal were ghostwritten by public relations agencies and paid for by pharmaceutical companies. Such articles have appeared in dozens of journals, and ghostwriting scandals have involved professors at the University of Pennsylvania, Brown and Harvard. Last year, court documents revealed that Monsanto ghostwrote articles that were published in peer-reviewed journals to counter research on the carcinogenicity of the pesticide glyphosate and to attack regulatory bodies.

Peer review has also sometimes stymied important research. Senior scientists are more likely to be asked to assess submissions, and they can shoot down articles that conflict with their own views. As a result, peer review can act as a shield to protect the status quo and suppress research viewed as radical or contrary to the established perspectives of referees. A 2015 study of 1,000 medical journal submissions found that of the papers that were eventually published, the 14 that became the most frequently cited were initially rejected. Groundbreaking studies by Sir Frank MacFarlane Burnet, Rosalind Yalow, Baruch Blumberg and others were rejected by peer reviewers, yet later led to Nobel Prizes.

Scientists understand that, despite its limitations, we need peer review: It is hard to imagine how we would get along without it. They also understand that it’s a human process and thus rife with human failings. But these subtleties are rarely communicated to the public. With more transparency about the publication process, we might have a more nuanced understanding of how knowledge is built — and fewer people taking “peer-reviewed” to mean settled truth.

Twitter: @thackerpd, @Protohedgehog

Read more from Outlook:

Science educators need to talk about the identity of scientists

Why scientists sometimes make extraordinary claims

Follow our updates on Facebook and Twitter.