The Washington PostDemocracy Dies in Darkness

Opinion White House science council: Bite-mark matching is junk science

(Julio Cortez/Associated Press)

Jordan Smith at the Intercept has the details:

The President’s Council of Advisors on Science and Technology (PCAST) has concluded that forensic bite-mark evidence is not scientifically valid and is unlikely ever to be validated, according to a draft report obtained by The Intercept. The report, titled “Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods,” is marked as a “predecisional” draft created August 26 that is not to be quoted or distributed, though the title page suggests the report will be made public sometime this month. . . .
PCAST finds that bitemark analysis does not meet the scientific standards for foundational validity, and is far from meeting such standards,” it reads. “To the contrary, available scientific evidence strongly suggests that examiners cannot consistently agree on whether an injury is a human bitemark and cannot identify the source of [a] bitemark with reasonable accuracy.” . . .
While the report acknowledges the dentists’ concern that excluding bite-mark evidence “could hamper efforts to convict criminals in some cases,” it rejects the idea that this alone is sufficient to warrant the practice. If the dentists’ concerns are valid, “the correct solution, from a scientific perspective, would not be to admit expert testimony based on invalid and unreliable methods, but rather to attempt to develop scientifically valid methods,” it reads. “However, PCAST considers the prospects of developing bitemark analysis into a scientifically valid method to be low. We advise against devoting significant resources to such efforts.”

This is about as strong as this sort of condemnation gets. The report also criticizes other “pattern matching” forensic fields with varying degrees of severity, including shoe-print matching, firearms analysis, tool-mark analysis and even fingerprint matching (which likely has more evidentiary value than most other pattern-matching disciplines but is often overstated and needs more research to determine accuracy and margin for error). Pattern-matching fields of forensics basically involve a self-proclaimed expert looking at a piece of evidence and declaring by his or her own judgment whether it excludes, is consistent with or “matches” a suspect. It’s highly subjective, and the most successful practitioners aren’t necessarily the most thorough or careful analysts, but the people who are best at persuading juries.

The fundamental flaw in these fields is that we just don’t know how unique these patterns actually are. Unlike with DNA, where we know how frequently certain markers appear in the population,  you simply can’t say that “this screwdriver and only this screwdriver made the pry marks on the victim’s door.” (Believe it or not, similar testimony has been used to win convictions in countless cases.) No expert can say say with any certainty how many screwdrivers could have made similar marks. Perhaps one could create similar marks with the screwdriver found in the suspect’s house. But one could probably also do it with thousands of other screwdrivers. Or chisels. Or crowbars.

The report (which I’ve also seen) is also critical of the use of multiple-source DNA samples. While DNA evidence from one source in, say, a rape case is objective and scientifically sound, things get tricky and much more subjective when you’re sorting through DNA from multiple sources. The use of “touch DNA” can involve a half-dozen to dozens of sources. The more sources of unique DNA, the more subjective the analyst’s role.

But let’s get back to bite marks. As Smith reports, the National District Attorneys Association (NDAA) has already preemptively attacked the report, calling it “scientifically irresponsible.” So Group A just put out a report criticizing certain forensic disciplines as unscientific. Group B just called Group A “scientifically irresponsible.” Which group should you believe?

Well, let’s think about that. Group A is made up of renowned scientists —  people who work in professions governed by objectivity; who have achieved great success in fields where data is of the utmost importance and rhetoric means nothing; and who have no personal stake in whether these particular disciplines continue to be used in the courtroom.

Group B is made up of district attorneys — people who work in an adversarial profession; who have achieved great success in a field where rhetoric and persuasion are just as if not more important than facts (remember, most DAs are elected); who work with individual cases, not with data; and whose ability to win convictions stands to get much more difficult should these disciplines be barred from the courtroom.

Group A’s leaders include people such as Eric Lander, a geneticist, biologist and mathematician at the Massachusetts Institute of Technology; Sylvester James Gates Jr., who studies superstring theory and particle physics at the University of Maryland; Susan Graham, acclaimed computer scientist at the University of California at Berkeley; and William H. Press, an astrophysicist, theoretical physicist, computer scientist and computational biologist at the University of Texas.

Group B’s leaders include people such as former Cook County, Ill., state’s attorney Anita Alvarez; Shelby County, Tenn., DA Amy Weirich; Philadelphia DA R. Seth Williams, and Robert McCulloch, the prosecutor in St. Louis County, Mo.

And then there’s the current president of Group B, Mike Ramos. From Smith’s article:

NDAA president Mike Ramos, the elected district attorney in San Bernardino County, California, said that the NDAA will “continue to serve the citizens of this great nation” in part by defending the criminal justice system “against those who would seek to undermine” it.
Notably, it was under Ramos’s leadership that Bill Richards was wrongfully convicted of his wife’s murder; his office tried Richards three times before finally getting a conviction, and did so only after the questionable bite-mark evidence was introduced. After the California Supreme Court in late May threw out that conviction, which it determined was based on junk science, Ramos’s office made it clear to defense attorneys that they would seek to try Richards again. Ultimately, Ramos’s office dismissed the case, and Richards was finally freed.

I’d think we should probably take our scientific advice from Group A, no?

As for Ramos’s hysterics, if the courts were to adopt the PCAST position on these forensic disciplines, it wouldn’t “undermine” the criminal justice system; it would vastly improve it. It would prevent future wrongful convictions and make future convictions more credible.

The only thing undermined would be the ability of prosecutors like Ramos to send people to prison (or the execution chamber) with fake science. Perhaps that means some guilty people will go free. If so, they’ll go free because the state didn’t have enough legitimate evidence to convict them in the first place. But if we’re going to put people in prison based on non-science masquerading as real science, we might as well allow tarot cards, palm reading and necromancy into court, too.

The downside to Wednesday’s news is that no court is obligated to adopt the PCAST recommendations. And this really gets to the heart of the problem.

Seven years ago, the National Academy of Sciences came out with a similar report, though it was somewhat more diplomatic than PCAST’s. There were predictions at the time that the NAS report would fundamentally alter the way forensics could be used in the courtroom. That never happened. The NAS report did pave the way to a series of federal working groups (such as PCAST) to look into these fields, but in the end, one report led to a bunch of commissions and committees putting out more reports. The effect on the day-to-day workings of the criminal justice system was negligible.

Real change would require that we stop making judges the gatekeepers of science in the courtroom. The judges who decide whether a new forensic discipline has the scientific chops to be presented to a jury are no more qualified to make that determination than the NDAA. Which is to say that like prosecutors and defense attorneys, judges are trained in law, not science. And because our criminal justice system values precedent and finality, once one court has allowed bad science into evidence, others tend to follow suit. Even when the scientific community later catches up and declares some discipline, field or diagnosis to be flawed or fraudulent, courts are loath even to stop letting the bad science into new cases, much less overturn old ones.

Bite-mark analysis is a perfect illustration of the problem. Bite-mark matching was never accepted in the scientific community. At best, real science just never took much of an interest in it. It was a discipline born and developed solely within the confines of the criminal justice system. Today, you’d be hard-pressed to find a bona -fide scientist anywhere who would vouch for the scientific validity of bite-mark analysis as it’s practiced (or really at all). And yet it has been accepted in U.S. courts for over 40 years. The 2009 NAS report was highly critical of the practice. And yet in the seven years since, not a single U.S. court has upheld a challenge to its scientific validity. (Some judges and appeals courts have finally expressed some doubts about it, but only as asides.)

The PCAST report is welcome news. But it doesn’t necessarily mean the courts will abide by it. And as the NDAA response indicates, prosecutors aren’t going to give this stuff up without a fight. Our best hope to undo the damage already done is that when it comes to science, our courts finally listen to scientists instead of lawyers. That means barring these fields going forward and revisiting the cases in which they’ve been used. But merely addressing bad forensic disciplines after they’ve been polluting the justice system for decades isn’t enough. We need a system that prevents quackery masquerading as science from ever getting into court — one that prevents new charlatans pitching new snake oil from being used in the future. Our best hope there is to start putting the important task of distinguishing good science from bad in the hands of trained scientists, not in the hands of people who have been trained to practice law.