So when a peer-reviewed study was published last year in the prestigious Proceedings of the National Academy of Sciences that purported to have overcome these obstacles, we read it with great interest. The study, which analyzed data on roughly 900 fatal police shootings, claimed to show that “White officers are not more likely to shoot minority civilians than non-White officers.”
But while the study was widely covered in the media, it was based on a logical fallacy and erroneous statistical reasoning, and sheds no light on whether police violence is racially biased. We demonstrated this mathematically in a letter published in the same journal last week, but only after months of public protest and controversy. The difficulty of fixing blatant mistakes in academic publications threatens not only the advancement of science but also the promise of evidence-based policymaking.
It takes no technical expertise to understand the core problem of the study. The authors used data on fatal police shootings to determine the likelihood of officers shooting minority civilians, ignoring the fact that most police encounters with people do not result in a fatal shooting. Under this fallacious approach, an officer who encountered one minority civilian and fatally shot him or her (a 100 percent fatal shooting rate) would appear identical to an officer who shot one minority civilian out of a thousand similar encounters (a 0.1 percent fatal shooting rate). Data on fatal shootings alone cannot tell us which officers are more likely to pull the trigger, let alone account for all relevant differences between incidents to allow us to isolate the role of race.
Given this glaring flaw, we anticipated that the record would be swiftly corrected. On the contrary, PNAS promptly rejected our critique last year. An editor at the journal told us the clear logical errors we highlighted merely reflected our “preferences” over how to best study this question, and that the tone of our critique was “intemperate.”
Determined to correct the record, we turned to a venue that has, for better or worse, become a primary locus of academic debate: Twitter. Our post was met with outrage, both from supporters of the study and other researchers who were perplexed as to how a leading scientific journal could allow such an obvious error to stand.
In response, the original article’s authors posted a reply to our critique. Though they still largely stood by their study, they admitted their central claim — that white officers are not more likely to shoot minority civilians than their nonwhite peers — was unsupported by their analysis. Still, that didn’t prevent the study’s findings from being widely disseminated, including in U.S. congressional testimony in September as evidence that policing does not suffer from racial bias.
We decided to appeal our rejection at PNAS — typically a long-shot strategy when it comes to academic publication. This time, to our surprise, our critique was accepted, with the caveat that the study’s authors also be given space in the journal to respond.
This resolution, and the bizarre and circuitous sequence that preceded it, reveals the often-arbitrary nature of the academic publication process. Not only did one of the most-cited scientific journals in the world allow a flawed study into print, it published criticism only after months of social media controversy. In addition, the journal has opted to publish our concerns in a “he said, he said” format opposite the original authors’ defense, likely sustaining public confusion.
In response to this op-ed, PNAS editor in chief May Berenbaum said in a statement: “As a longstanding policy, PNAS attempts to coordinate the publication of Letters to the Editor with responses from the authors of the original articles to ensure that readers have access to the full range of perspectives and supporting arguments, thereby providing a broader context in which to evaluate the critiques and responses.”
To be clear, we are well aware that the process of scientific discovery is often messy, and passionate debates between scholars who disagree are a vital and necessary part of the furtherance of knowledge. These debates often live in a gray area in which the arguments of both sides have some merit, with no clear right or wrong answer immediately to be found.
This is not one of those cases. If scientific journals fail to label mathematically flawed assertions as wrong, opting instead to equivocate, the general public cannot be blamed for basing its opinions on false premises.
In many ways, the current era represents a golden age for the scientific study of social problems. New high-resolution data and sophisticated techniques for parsing correlation from causation are allowing unprecedented insight into human behavior. And politicians have actually expressed interest in evidence-based policy, leaving the door open for academics to make a real-world impact.
But the promise of this moment will be wasted if scientists cannot be relied upon to separate fact from fiction. We must do better, or risk receding into justified irrelevance.