In September, the President’s Council of Advisors on Science and Technology (PCAST) issued a scathing report on the use of forensic analysis and expertise in the criminal-justice system. The report, “Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods,” looked at pattern matching forensic disciplines such as bite mark matching, shoe print matching, blood spatter analysis, fingerprint matching and hair fiber analysis. It also looked at DNA testing when investigators find biological material from multiple sources, a scenario that can bring human subjectivity into the testing. With the exception of single-source DNA testing, the report found serious deficiencies in all areas of forensics it studied.
The PCAST report was damning, but if you’ve been following these issues with any regularity, it wasn’t at all surprising. That was in September. It’s now January. And not only has the Obama administration done nothing about the report, the Justice Department has publicly denounced it. That report, along with others and an administration that seemed unusually equipped to take it seriously, presented a small window in which to reform a system. That window is about slam shut. And we’re about to be governed by a new administration that seems likely to board it up, wallpaper it and overlay it with brick. This wasn’t just a missed opportunity; it was a catastrophe. And it’s difficult to overstate the consequences.
Nearly all the fields PCAST studied were in the field of pattern matching, in which an analyst looks at evidence from the crime scene and either draws conclusions about how the crime occurred or attempts to “match” that evidence to a suspect or suspects. The problem is that all of these fields are inherently subjective. They rely not on scientific analysis or mathematical computations but on the judgment of the analysts. This is why while you’ll frequently see two bite mark analysts or blood spatter analysts give mutually exclusive testimony in a case, you’ll rarely if ever see it among two DNA scientists (at least when the tested DNA came from a single source).
Inevitably, then, the amount of weight a jury typically gives to evidence from these fields of forensics depends far more on the persuasiveness of the expert witnesses than on the evidence itself. In fact, there’s a strong bias in favor of testimony that’s less scientific. In recent years, prosecutors and defense attorneys alike have called attention to the “CSI Effect,” the way the popular TV franchise has conditioned jurors to look for forensic evidence that’s definitive and foolproof. Most forensic evidence isn’t completely clear cut. It’s merely suggestive. So a forensic analyst who gives testimony properly grounded in science will be careful not to speak in absolutes, will use cautious and careful language — and will be far less successful at persuading juries. The analysts who gleefully gallop past scientific boundaries in a rush to implicate suspects are far more successful in the courtroom. Because prosecutors and defense attorneys want to win cases, many will opt for the analysts who give them the best chance to do so. The quacks rise to the top.
There are other problems, too. With the exception of DNA testing and to some extent forensic pathology, most forensics fields were invented and developed by people with backgrounds in law enforcement, not in science. This is why crime labs, which claim to be scientific in nature and method, typically fall under the auspices of state police agencies, offices of the attorney general, departments of public safety and other law-enforcement-oriented bureaucracies. So not only has forensics grown less scientific over time, it was never grounded in science to begin with. DNA was discovered by scientists. Consequently, we know the frequency with which certain DNA markers appear across the human population. This is why a DNA analyst can give precise calculations of the odds of the suspect being the person whose DNA was left at the crime scene or in the rape victim. (Again, provided we’re talking about cases in which there was only one source for the DNA.) Put a different way, DNA analysts can provide an error rate — a calculation of the chances that the match is mistaken. (It’s usually an extremely low number.)
Pattern matching analysts can’t do that. When analysts look at the characteristics of hair fibers to distinguish one from another, we don’t know the rate at which those characteristics are distributed across the population. The same goes for distinguishing the features analysts look for in bite marks. It’s even true of fingerprints.
With many pattern-matching fields, the material in or on which the data is recorded presents an added problem. Even if we could somehow prove that everyone’s bite is unique, there’s no scientific evidence to support the notion that human skin is capable of recording those unique features in a usable way. (The existing research strongly suggests the opposite.) There are just too many variables. The victim may have been pulling away as the bite was inflicted. The wound could have been corrupted by heat, cold, insects or any number of other things. People heal differently, and the elasticity, sponginess and toughness of skin varies from person to person, and even on different parts of the same person. There’s the angle at which the biter struck, whether the biter drug his teeth or bit straight down, and so on. The same goes with tire or shoe marks in dirt or mud, blood spatter patterns and so on. There are too many variables that could affect how these patterns are recorded and preserved.
Nevertheless, these fields have been used over and over in court. Because these disciplines were developed outside the field of science, and because science and law tend to mix like oil and water, it has taken a while for scientists to catch up, and to test the claims of forensic analysts. It was really DNA testing that began to expose the flaws in forensic analysis, as testing began to overturn convictions that forensic expert witnesses had testified were rock-solid.
The aim of the PCAST report, as well as the 2009 National Academy of Sciences report and the National Institute of Standards and Technology working groups, was to bring science to forensics — or rather to see if there’s any science behind the claims of forensic analysts. The 19 PCAST members are all scientists. While some legal experts did provide some guidance, the evaluation itself was done by some of the most eminent scientists in the country. These weren’t defense attorneys or social justice activists.
What PCAST, NIST and the NAS have found, overwhelmingly, is that all pattern matching fields of forensics lack any scientific support for their basic assumptions. The Obama administration seemed to understand these problems like no administration to date. Obama himself along with former attorney general Eric Holder have openly acknowledged that the criminal-justice system is flawed and in need of reform. In July 2015, Jo Handelsman, the assistant director of the White House Office of Science and Technology Policy, called for the eradication of bite mark evidence in court. It’s the Obama administration that oversaw the creation of the NIST working groups to study the validity and reliability of various forensic disciplines. And it was obviously Obama’s own PCAST that produced the paper last year in the first place. The White House even put out a news release touting the report.
And yet when the PCAST study came out, Attorney General Loretta Lynch immediately dismissed it, stating that while “we appreciate their contribution to the field of scientific inquiry, the department will not be adopting the recommendations related to the admissibility of forensic science evidence.” Lynch joined the National District Attorneys Association and various police organizations in casting the study as no big deal.
There are a number of reasons why it is in fact a big deal. First there’s the FBI, which Lynch oversees. Obviously, if the FBI crime lab isn’t going to make changes in the face of these reports, that’s going to affect the quality of the forensic evidence used in federal criminal cases. But the FBI crime lab also does forensics work for local police and prosecutors across the country, so those cases will also be affected. FBI crime lab analysts train state and local analysts, so the bad methods are passed on. The FBI crime lab is also generally seen as the preeminent crime lab in the country, so other crime labs emulate it. If the FBI isn’t going to change in the face of overwhelming scientific evidence, why should anyone else?
Then there’s the matter of the FBI’s own record. In her statement in September, Lynch argued that the FBI doesn’t need to abide by the PCAST recommendations because, as the Wall Street Journal put it, “the Justice Department had taken unprecedented steps to strengthen forensic science, including investments in research, draft guidance to lab experts when they testify in court and ‘reviews of forensic testimony in closed cases.’”
But despite Lynch’s assertions, and despite its reputation, the FBI has a long and sordid record when it comes to dubious forensics dressed up as science. For more than 30 years, for example, the FBI advanced a theory that every batch of lead used to manufacture bullets contained a unique chemical fingerprint. If a bullet found at a crime scene had the same chemical signature as a box of bullets found at a suspect’s home, the bullet could have been manufactured only with those other bullets, which would strongly implicate the suspect. So-called “Compositional Bullet Lead Analysis” has since been disproved, but even after it was shown to be false, the federal government vigorously fought against releasing the 2,500-plus cases in which the analysis was performed. As my Post colleague Spencer S. Hsu has shown in a series of articles, FBI hair and fiber analysts too vastly overstated their findings in hundreds of cases covering several decades. Worse, those FBI analysts then trained countless analysts at state and local police agencies across the country to use the same flawed techniques. It’s impossible to say how many cases were possibly tainted, but it’s likely well into the thousands. Here, too, the government for years made no attempt notify the people who may have been wrongly convicted. Those are just two examples. There are plenty of others. And scandals at other crime labs across the country have grown too numerous to count.
Another criticism of the PCAST report came from the National District Attorneys Association, which complained that the committee that authored the report included “not a single working forensic scientist, and this lack of scientists with real-world experience is shown throughout the report.”
This is nonsense. If you were to commission a report on the scientific validity of homeopathy, or astrology, or creationism, you wouldn’t ask homeopaths, astrologists or creationists to serve on the committee that researches and authors the report. Forensic analysts want to be called “scientists,” but they don’t want to subject their methods and conclusions to the scientific method. They don’t want their work to be evaluated by “other scientists.” They can’t have it both ways.
Furthermore, the PCAST report doesn’t call for the elimination of all forensics. It merely states that when an expert witness presents testimony to a jury, there should be scientific research to support what that expert is saying, and that all fields of forensics should be subject to the tools of basic scientific inquiry, like peer review, blind testing and proficiency testing. This apparently is too much for Lynch and the FBI. Despite the poor track record of the latter, Lynch’s position here seems to be: Just trust us. We have it all figured out now.
That isn’t close to good enough. Even if the FBI has occasionally stopped using some methods subsequently shown to be wholly invalid, new forensics disciplines are emerging all the time. (Look up “touch DNA,” “shoe forensics” and “forensic linguistics.”) Until there are rigorous scientific standards for determining which of these new fields are legitimate and which aren’t, we’re always at risk of the next “voice print” technology or bite mark analysis corrupting the courts. (Given that courts still haven’t upheld a challenge to the latter, the point seems indisputable.)
Over much of President Obama’s tenure, he has said some important and, in some cases, historic things about criminal-justice reform. He deserves credit for that. But too often those statements have been followed by policy changes that have been largely symbolic. It’s an issue upon which he hasn’t been willing to expend much political capital for substantive change.
But the forensics problem doesn’t fit even that pattern. It’s much worse. Here’s why:
As I wrote above, it really took DNA testing to convince the courts, politicians and much of the public that the criminal-justice system was in need of repair. For decades, the country (or at least the parts of the country unlikely to be caught up in the criminal-justice system) plugged along thinking that our system gets it right all the time, or very nearly all the time. DNA testing changed that. But as I’ve written here before, DNA testing isn’t a panacea; it’s a wake-up call. The percentage of all cases in which DNA testing conclusively proves or disproves guilt is small — 10 percent at most. But the flaws in the system that DNA exposed in those 10 percent of cases almost certainly persist throughout the system, and likely at about the same rates. Today, provided that it’s done correctly (by no means a given, because human beings will always be doing the collecting), DNA testing can make us reasonably certain that the system will get it right nearly all the time for that 10 percent of cases. But if we don’t correct the problems DNA testing originally exposed in those 10 percent of cases, those problems will continue to plague the remaining 90 percent of cases.
Even today, while most appellate courts will give innocence claims based on DNA test results strong consideration, anything short of a DNA test still faces a very high bar. Here at The Watch, we’ve looked at a number of cases in which courts have refused a strong innocence claim in a case for which DNA wasn’t relevant, even though the defendant was convicted by forensic techniques that DNA testing has already exposed as dubious in other cases (see bite mark matching). In some instances, courts have refused claims by defendants convicted by testimony from experts whom an appellate court itself has previously acknowledged to be unreliable. Without DNA, it’s just really difficult to get a conviction overturned.
Here’s the haunting part: It’s unlikely that a technology as revolutionary as DNA testing will ever come along again. If we don’t fix the structural problems that DNA testing has exposed, not only will those problems continue to plague the other 90 percent of cases — without those occasional definitive wrongful convictions to grab us by the lapels — courts, politicians, and much of the public will over time be lulled into a new but just as false faith in the system. Without more DNA exonerations, we’ll come to believe that DNA fixed things. And without more DNA exonerations, it’s a false faith from which it’s unlikely that we’ll ever be jarred loose. The wrongful convictions will continue, but they’ll continue with an ever-shrinking possibility of ever being discovered. DNA testing wasn’t just a wake-up call. It was a one-time wake-up call. We ignore it at our peril.
And this is why Obama’s acquiescence to Lynch and the FBI is so maddening and utterly disappointing. This is an administration that claims to believe in science. The science here isn’t in dispute. It is clear and overwhelming. This is an administration that claims to care about justice. The injustices here aren’t in dispute. They are real and thoroughly documented. And they will almost certainly continue.
So what should Obama have done? He should have discussed the PCAST report with Lynch before she issued a public statement. If she refused to implement the report’s recommendations, he should have asked for her resignation. The same should have been true for any heads of the federal law enforcement agencies who ignored the report. This is that important. Forcing someone such as James Comey to resign may have presented a political challenge, but at the very least, Obama could publicly chastise him for his obstinacy. He could have forced him to publicly defend his actions. Perhaps he’d have spurred some congressional hearings.
Yes, it’s true that if Obama and Lynch had implemented the PCAST recommendations, the Trump administration and a Jeff Sessions-led Justice Department would still likely try to undo them. Of course, Obama and Lynch didn’t know last fall that there would be a Trump administration. And in any case, so what? That’s true of most executive policies. It would at least have created a record. It would at least have forced Sessions to undo it, and then to explain why. As it stands now, should anyone decry the FBI’s disregard of the scientific method in its use of forensics, Sessions will be able to argue that even the activist Obama administration (this would be his characterization, not mine) dismissed such complaints out of hand.
Prominent Democrats should be reprimanding Obama and Lynch for this. Democrats like to tout themselves as the party of science, usually with a dose of smug (and sometimes justified) condescension. Multiple federal reports authored by preeminent scientists have now emphatically stated for years that bad science is corrupting the justice system. Among real scientists, there’s little dispute about this. The first of these reports came out in 2009. Obama has had seven years to do something substantive about it. He hasn’t. The Democrats also like to tout themselves as the party of justice and equality. The people most affected by these problems are disproportionately poor and disproportionately black. And yet … nothing.
If Obama was just going to allow the DOJ to ignore the PCAST report all along, what was the point of commissioning it in the first place? Why call in the science if the science won’t change anything? The stakes couldn’t be higher here. And the demand is simple: If federal police and prosecutors should want to use expert testimony to put people in prison (or execute them), that testimony should be subjected to and governed by the basic rules of scientific inquiry.
If the Democrats can’t muster the political will for even that, they need to stop claiming to be the party of science.