(Bigstock)

There’s been much talk this week about a new study from Harvard economics professor Roland G. Fryer Jr. on racial bias in police shootings. Much of the coverage has focused on the study’s surprising-to-some conclusion that racial bias doesn’t factor into police use of lethal force, at least in the city of Houston and at least once the officer has stopped a civilian.  There have also been some interesting critiques from (among others) Rajiv Sethi, Ezekiel Kweku and Dara Lind. But the most pertinent flaw in the study (which Fryer has tried to explain, I think unsatisfactorily) is the same flaw in any study that relies on police reports: It relies on police reports.

We want to reform policing. But we want those reforms to be informed, based on good data. The problem is that nearly all the data we have on incidents involving police officers using lethal force comes from reports written by police officers, and nearly all of those reports were written by the officers who were actually involved in those incidents.

The current law on when police officers may use lethal force allows for what critics (like me) would say is far too much discretion. It doesn’t account for police officers who needlessly escalate a situation and then have no choice but to use lethal force due to the circumstances they created. It doesn’t account for mistakes made by police officers themselves that might have caused an officer to reasonably believe a suspect posed an imminent threat. It doesn’t account for police officers giving contradictory commands, then shooting someone for misinterpreting them.

For the purpose of the discussion, let’s break shootings and killings by police into three categories: incidents that were illegal and unnecessary, incidents that were legal and necessary, and incidents that were legal but unnecessary. If you’re asking whether current laws and policies allow for too many police shootings, looking at how many shootings are justified under current law and policy is just question begging.  It’s that last category — legal but unnecessary — that we want to explore. Unfortunately, it’s also a category that is plagued by subjectivity and the simple fact noted above: Most of the data we have comes from police reports themselves.

If we were to compile statistics on, say, medical mistakes in an effort to make policies that would improve the state of medicine, we wouldn’t get all of our data from written statements by the accused doctors or hospitals. If we wanted to compile data on conflicts of interest in politics, we wouldn’t rely on politicians to self-report and adjudicate when their vote may have been influenced by a campaign donation. But this is essentially what we do with shootings by police officers.

People in Dallas, St. Paul, Minn., and across the nation grapple with how to move on from the deaths of two black men at the hands of police as well as the loss of five police officers all in one week's time. (Whitney Shefte,McKenna Ewen,Dalton Bennett,Ashleigh Joplin/The Washington Post)

The argument here is not that there’s something uniquely untrustworthy about cops. The argument is that almost every police officer who has just shot and killed someone will defend his or her decision to kill. It’s human nature. It could be because the killing was entirely justifiable. It could be because the officer wants to believe it was justifiable. It could be because the officer knows it wasn’t justified, but fears the consequences.

Personally, I suspect that a high percentage — well more than half — of shootings by police are both legal and justified. I also suspect that nearly all cops who have just shot and killed someone truly believe that their actions were justified. That is, I suspect that the percentage of cases in which cops knowingly covered up a bad shoot is pretty small. But I also think it’s safe to say that the percentage of shootings by police that most of the public would find troubling, unnecessary or unjustifiable is far below the 99 percent or higher of cops that are cleared in these cases.

But the preceding paragraph is loaded with hedges like “suspect” and “think” for a reason. On a macro level, these details are almost unknowable. And that’s a huge problem.

In too many cases to count now, surveillance video, dash cam footage, body cam footage or citizen-filmed video has shown that a shooting didn’t happen the way the police officers involved claimed it did. The charitable explanation for the discrepancy is that the police simply misremembered what happened. That is an entirely human reaction. The less charitable interpretation is that the officers knowingly lied. That’s also human, though obviously not acceptable. For this discussion, the difference isn’t all that important. What’s important is that without video, the officer’s account is usually all that matters. But we know from video that the officer’s account can be wrong. We don’t know how often this is the case. And it isn’t clear that this really can be known.

For the purpose of collecting statistics, even when video contradicts the officer’s account, the officer’s report is usually what’s sent to the FBI. (It’s actually worse than even that: Up until only recently, the FBI only tracked justifiable homicides by law enforcement. In the rare instance that a shooting wasn’t legally justified, it wasn’t even included in the data.) But if we know from these video cases — and just a knowledge of human nature — that the police reports are sometimes inaccurate, that they will nearly always be inaccurate in a way that supports the police, and that it’s difficult if not impossible to know just how often they’re inaccurate, how much should we rely on the statistics we draw from those reports?

Already we’re seeing that the more data we have, the less certain we are about how often shootings by police were justified. (I use justified in a general sense here, not a legal sense.) According to 2014 report in the Wall Street Journal, over a seven-year period ending in 2011, 41 police officers were criminally charged for an on-duty shooting. Over the same period, the FBI reported 2,711 justifiable homicides by law enforcement. That means that in at least 98.5 percent of police shootings, the officer was cleared of criminal wrongdoing. Given that we now know that the FBI figures have been vastly underreporting police shootings, it seems safe to say that the clearance rate is over 99 percent.

According to The Post’s data for 2015, in “74 percent of all fatal police shootings, the individuals had already fired shots, brandished a gun or attacked a person with a weapon or their bare hands” — what the project has deemed an “attack in progress.” This has led many to conclude that 74 percent of police shootings are justified. The Post’s figure draws on media reports in addition to police reports, so it’s significantly more nuanced and targeted than the FBI data. It’s also already a much lower number than the percentage of shootings deemed justified by judges and prosecutors. Note that the Post doesn’t state the other 26 percent were illegal, just that they weren’t clearly justifiable, so it isn’t a perfect comparison. But this still means there are significant questions about one in four fatal police shootings. Before, all we had to go on were prosecution statistics, which cleared cops in more than 99 percent of shootings. More information has raised more questions. I don’t know what an acceptable number would be, but one in four seems awfully high.

But even that 74 percent figure needs more examination. Look at the shootings of Philando Castile and Alton Sterling, the high profile cases that have spurred the most recent round of protests. In both cases, the deceased was in possession of a gun. In both cases the police claimed that the suspect was reaching for that gun. For FBI data-collecting purposes, both cases would be instances in which police officers used lethal force to defend themselves after an armed suspect went for his gun. Even with the videos, that’s how they’d be counted.

But it’s clear that after being presented with the details of these shootings, a good percentage of the public consider them somewhere between troubling and outrageous. It also seems clear that without those videos (both shot by citizens), few would have questioned either shooting at all.  I asked The Post’s Wesley Lowery about the two cases. He said that the Post’s project would likely count both incidents as having an “undetermined” threat, but only because we have video. Without the video, The Post would likely have classified them as an “attack in progress.” That should at least have us asking questions about what other cases among that 74 percent we would be rethinking had there been video.

I can think of several cases we’ve covered here at The Watch that illustrate this problem:

  • Lori Jean Ellis was gunned down in her own home when three police officers showed up in the middle of the night to serve a series of warrants for petty crimes. The police officers claimed they killed Ellis when she fired at them with a “high-powered rifle,” and claimed to have seen a muzzle flash, heard a loud boom, and witnessed smoke coming from the barrel. In truth, the only gun in Ellis’ home was a pellet gun, and the crime lab couldn’t even get it to fire. Yet for the purposes of data collection, Lori Jean Ellis would be listed as an armed suspect who fired a gun at police.
  • Ernest Russell, Jr. was also killed in his own home during a misdemeanor gambling raid. The police claimed to have warned Russell over a dozen times that they were cops, and that he should drop his weapon and raise his hands. But police body cam video shows that Russell was killed less than five seconds after the first officer entered his home. They did find a handgun near Russell’s body, but it hadn’t recently been fired, and even lacked any of Russell’s trace DNA. Between the cowboy, unprofessional raid tactics, the disproportionate use of force for a misdemeanor crime, the fact that Russell was taken by surprise, and the vast chasm between the police report and the body cam video, I think most people would consider Russell’s death at the hands of police to unnecessary and preventable. Many would probably consider it an outrage. Yet for purposes of collecting statistics, Ernest Russell, Jr. too would be a suspect who was killed after confronting police officers with a gun.
  • The same goes for Julian Betton. He was nearly killed when an anti-drug task force battered down his door over a couple of $50 pot sales to an informant. The police initially claimed Betton fired at him. Crime lab testing showed his gun hadn’t been fired. The police claim they loudly knocked and announced themselves several times before battering down Betton’s door. Surveillance video shows they never knocked. The video lacks sound, but it indicates that even announcement was unlikely, several would have been impossible. The video clearly shows that the police violated the knock and announce rule. Betton survived (barely). But had he died, he too would have been an armed suspect killed by police after confronting them with a gun. The fact that the police broke the law and battered down his door over a small amount of pot wouldn’t matter.
  • Jason Wescott was also killed in his own home after an informant told police there was pot inside. Wescott had actually been confronted months earlier by an armed intruder who had previously threatened him. Wescott’s family says that when he reported the incident to police, they told him to get a gun and be prepared to use it if the intruder returned. Instead, it was the police who broke into Wescott’s home. When he reached for his gun, they killed him. They found about $5 worth of marijuana. It was later revealed that the informant whose tip was the sole basis for the raid had a history of lying. The informant himself later said the police had prodded him to exaggerate in order to secure probable cause for the raid. I think most people would find Jason Wescott’s death to be unnecessary and preventable. Many would find it outrageous. But for statistical purposes, he too would be an armed suspect killed by police officers after confronting them with a gun.
  • Marcus Cass was shot and killed by police as they raided his pawn shop for what was basically an administrative crime. A federal appeals court determined that Cass likely thought the store was being robbed, and chastised the police for the way they handled they raid. But for statistical purposes, Cass was killed after confronting police with a gun.
  • Gonzalo Guizan was killed during a volatile police raid on the home of Ronald Terebesi after a prostitute reported Terebesi had been using cocaine. Guizan was visiting at the time of the raid. Police claimed Guizan, who was unarmed and had no criminal record, attacked them as they broke into the house and threw flash grenades. It seems highly unlikely that an unarmed man with no record decided to attack a heavily armed team of raiding cops. It seems far more likely that Guizan mistook them for criminal intruders, and was trying to get out. The courts refused to grant the officers immunity in lawsuits filed by both Terebesi and Guizan’s estate. Yet for statistical purposes, Guizan would be counted as a man who was killed after attacking police officers.

These stories are just from the last year or so (or at least just came to light or were resolved in the last year). But I could run off a long list of other shootings from previous years. There have been countless drug raids in which a suspect pulled a gun on police, but in which it’s entirely plausible that the suspect had no idea the armed intruders in his or her home were police. In some cases, the police were in the wrong house. No matter: For statistical purposes those were armed suspects. I think of the Jonathan Ayers case, in which a drug task force, armed and dressed in undercover garb, ran at an innocent pastor as he sat in his car at a convenience store. The terrified Ayers sped away in his car, lightly brushing one of the officers in the process. They opened fire and killed him. In the police report, he attacked them with a deadly weapon — his car. I think of the Kathryn Johnston case, in which the police invented an informant and lied on a search warrant affidavit before breaking down the 92-year-old woman’s door. She was innocent. When she met them with the broken old revolver she used to scare off intruders, they shot and killed her. In the police report, she was an armed suspect who threatened them with a gun.

We’re definitely learning more about police shootings. Prior to projects like the one run by my Post colleagues, the journalists at the Guardian, and the Fatal Encounters website, we didn’t even know how many people U.S. police killed each year. We had to rely on FBI data, which depended on voluntary cooperation from police agencies. We now know that the FBI data severely undercounted even this basic statistic. Taking the work of the Post, Guardian, and Fatal Encounters a step further, Dallas-area detective Nick Selby and his co-authors added yet more context to these shootings, scanning police reports and press accounts to find information like whether police first tried to use non-lethal force, or if the suspect was suffering a mental health crisis.

But even these new databases can’t possibly tell the whole story. The crucial details that cast the killings of Lori Jean Ellis and Ernest Russell, Jr. in a completely different light, for example, were buried in mountains of police files, and only came to light years after the shootings. Most police shootings in South Carolina (where both shootings took place) are investigated by an outside police agency called the South Carolina Law Enforcement Division, but that agency’s investigators never followed up on crucial details, such as the fact that Ellis’ “high-powered rifle” was actually a pellet gun, and the evidence strongly suggests she was nowhere near her when she was killed. The damning video that contradicts the police account of the Russell shooting was uncovered by an attorney for Russell’s family years later. He got it from the cop who was wearing the camera at the time. The discrepancy between the video and the officers’ statements was never mentioned in the SLED report. Incredibly, the video itself was entrusted with one of the officers involved in the raid.

The point is, it’s nearly impossible to know all the details behind all of these shootings. We have to rely on reports filed by the officers themselves. We know more details about these particular cases because an attorney or a journalist took the time to investigate, file open records requests, and look beyond the police reports and press accounts (which, less face it, too often are too reliant on the police reports).

Last year, The Post counted 986 fatal police shootings. The Guardian found more than 1,100 total deaths at the hands of law enforcement. Even if we could somehow get all objective details about all deaths at the hands of police, determining how many of these deaths were preventable will still be highly subjective — probably too subjective to come up with guidelines to classify them in a meaningful way. We can all agree that an unarmed suspect shot and killed after ignoring a police officer’s commands and reaching for his waistband is tragic and awful. (Let’s assume that in most of these cases, the suspect really was reaching for his waistband, for whatever reason.) But was it unnecessary? Should that cop be punished? Could better training have prevented it?

Whether or not you think the officer shot too quickly in such a scenario likely depends on your politics, your personal experience with police officers, your ethnicity, and any number of other factors. The point is, we can’t start talking about what the data mean if there’s sincere but irreconcilable disagreement about how to classify the incidents before we compile them to create data.

I don’t mean to be fatalistic about all of this. It’s unquestionably a good thing that we have more data on police shootings than we did just a couple years ago. More information is always better than less. But we should be careful when drawing conclusions from that data, and even more careful when making policy based on those conclusions. For example, from the FBI data (which, as noted, is already unreliable) it appears that shootings by police are going up, even as shootings of police are going down. Does that mean cops are getting more violent, and unnecessarily so? It could, and I think there’s other evidence to suggest that police officers are more quick to use lethal force than in the past. But it could also be that fewer cops are dying because police today are more willing to use lethal force. Perhaps they’re killing the criminals before the criminals can kill them.

Of course, even if that is true, it could still also be true that too many shootings by police today are unnecessary. In other words, cops may be preventing police deaths by being quicker to use lethal force — but they also may be overcorrecting, resulting in unnecessary fatalities.

Comparing shootings by police to the crime rate suffers from similar problems. If shootings by police have gone up even as the crime rate has dropped, does that mean cops are unnecessarily killing people? Or does it mean more aggressive policing is helping to fight crime? Maybe there’s no relationship at all?

As Phillip Bump demonstrated two years ago, there are numerous ways to analyze shootings by police, and countless crime-related datasets to which we can compare them. Far less clear is what conclusions we should draw from those comparisons.

So what’s the alternative? There really isn’t one. Unless we’re going to assign data collection monitors to follow officers around, police agencies are the obvious primary source for this data, and really the only option for recording it in a comprehensive manner. The recent efforts by media and activist groups to complete, improve, and contextualize what police agencies report have brought us a long way, but they’ve also made it depressingly clearer just how much we don’t know. The Post alone has 60+ people working on its project, yet that’s not nearly enough personnel to delve into each shooting the way, say, South Carolina attorney Robert Phillips delved into the deaths of Lori Jean Ellis and Ernest Russell.

Again, none of this is to say this data is completely useless. We just need to be really cautious about how we use it, and realize that the numbers alone don’t always tell the story.