Deputy Attorney General Rod J. Rosenstein on Capitol Hill in June. (Alex Brandon/Associated Press)

In a speech at the International Association for Identification on Monday, Deputy Attorney General Rod J. Rosenstein announced the creation of a Forensic Science Working Group within the Justice Department. Recall that one of the first moves by Attorney General Jeff Sessions was to end the National Commission on Forensic Science (NCFS), which brought in independent scientists to evaluate the credibility of forensics fields used in U.S. courtrooms.

The new group will be housed within the Justice Department, which will inevitably make it less independent and less transparent than the NCFS. It will be led by Ted Hunt, a longtime prosecutor. Though Hunt worked with the NCFS, putting a prosecutor in charge of forensic reform doesn’t inspire a lot of confidence in the new working group’s impartiality.

For that matter, neither does Rosenstein’s speech. It’s been striking to see prosecutors and law enforcement groups react to the critiques of forensics that have come from the scientific community. Those critiques didn’t come out of nowhere. They came from commissions and panels formed after DNA testing, crime lab scandals and exonerations showed us that for decades prosecutors have been presenting juries with “scientific” evidence that is anything but. The FBI’s hair fiber analysis scandal alone tainted thousands of cases. And yet the reactions from prosecutors to these critiques — that experts are overstating their findings, wrongly implicating suspects, or practicing fields utterly lacking in any scientific principles at all — have been utterly devoid of any humility.

Rosenstein’s speech Monday is less strident than others I’ve seen, but it’s still preachy and didactic.

Those disciplines have been around for a long time. When subjected to informed cross examination, expert testimony can be tremendously probative and helpful to the jury.

Nevertheless, some critics have sought to limit the forensic evidence and testimony that can be presented in court. These critics suggest that unless a forensic discipline has a “known error rate,” evidence derived from that discipline should not be admitted in court. Under that standard, trusted and reliable forensic evidence would be excluded simply because the discipline is not susceptible to an easy-to-calculate error rate.

The folly of that approach is clear when critics question fingerprint analysis. They admit that it usually works. Their objection is that it requires judgment.

This isn’t quite true. Few are suggesting fingerprint evidence be barred from the courtroom. The criticism is that experts have been overstating the certainty of fingerprint evidence, particularly when it comes to partial prints. Fingerprinting has long been touted as the “gold standard” in forensics. It’s been touted as foolproof and definitive. The criticism here is that if we don’t actually know how unique a fingerprint is in a given population, we can’t tell jurors that they’re unique. If prosecutors want to present fingerprint evidence, jurors should be made aware of this. And if experts want to testify about “matches” and levels of “certainty,” let’s give them regular competency tests arranged by outside entities.

More from Rosenstein:

Evidence is relevant if it tends to make a material fact more or less probable than it would be without the evidence. It is not necessary for the evidence to be indisputable.

For example, a shoeprint with irregular edges and unique wear on the outsole found at the scene of a burglary is likely relevant and admissible. Both the prosecution and the defense can use the evidence to help the jury decide whether to believe a defendant was at the scene of the crime.

Physical evidence may be more helpful to a jury if an expert can explain the evidence and place it in context. The jury may benefit from expert testimony in interpreting how probative the shoe print is. Is it the same size as the defendant’s foot? Does it match a shoe found in the defendant’s closet? Answers to those questions may determine whether the physical evidence is incriminating or exculpatory. If the testimony were excluded, the search for truth would be impeded. The jury would have no assistance in determining what to conclude about the discovery of the shoeprint.

There’s nothing wrong with an expert stating that a shoe print is similar in size to the foot of a suspect or that it appears to be of a similar brand of shoe. But how similar? What if an expert tries to claim not that the suspect’s shoe is of a similar size and brand to those found in dirt near the crime scene but that he can tell by the wear in the tread on the suspect’s shoes that only that particular pair of shoes could have left the marks at the crime scene?

The problem is that there are no standards for making these assessments, nor for the language analysts use to convey those assessments to juries. And here we get back to error rates. With DNA, with blood typing and with other science-based fields of forensics, we can give juries probabilities. In the fields of forensics known as “pattern matching,” there are no such calculations to be made, because those fields are little more than experts relying on their expertise to “eyeball” the evidence. And when you’re eyeballing it, it becomes pretty easy to start to seeing matches that aren’t matches at all.

More from Rosenstein:

When the judicial system functions as intended, justice is advanced. Our adversarial system is based on the principle that the truth is most likely to emerge when opposing parties have the opportunity to cross-examine each other’s witnesses, and each party is able to call its own witnesses and introduce conflicting evidence.

Our criminal justice system provides important procedural protections for defendants. For example, a federal defendant is entitled to a written summary of the testimony of the government’s forensic expert. That allows the defendant to know in advance what the government’s expert plans to say, and to rebut it with his own expert testimony. The defendant also has the right to examine and challenge the results of any scientific test.

The search for truth benefits from those protections. But most of all, the quest for truth benefits from prepared legal practitioners and trained forensic examiners.

This cuts to the heart of our adversarial justice system. It also cuts to the heart of what’s wrong with it. The very fact that two experts claiming to employ a form of scientific analysis can testify in direct opposition to one another about, say, a bite mark or blood spatter or a hair fiber, is a pretty good indication that the field isn’t all that scientific. You’ll rarely, if ever, see two DNA experts at odds over how many markers match between two samples.

There are a few reasons why we can’t simply rely on cross-examination and our adversarial system to weed out bad expert testimony. The first is that when two experts contradict one another, the jury will be won over by the expert who is most effective at persuading juries. And that isn’t always the expert who is using the best science. In fact, as prosecutors themselves often admit, juries love certainty. They love experts who will tell them what to think. Scientists tend to avoid speaking with certainty. They speak in probabilities. All of which means the adversarial system may be biased against more scientific evidence.

The second is that we have relied on the current system for most of our history. And DNA testing has revealed that doing so has sent a lot of innocent people to prison. Rosenstein’s Justice Department oversees the FBI. The FBI used flawed hair fiber and ballistic evidence for decades. All of those experts were subject to cross examination. The FBI now admits that its experts were wrong. Should the FBI be allowed to continue to have experts testify to these flawed theories, on the theory that defendants will get to cross examine them? Of course not. The flaws in the agency’s hair fiber and ballistics analysis were exposed with competency testing, statistical analysis. So why not ask the scientific community to subject a field of forensics to that sort of scrutiny before it’s used to send thousands of people to prison instead of after?

More from Rosenstein:

The Department of Justice has undertaken unprecedented efforts to examine and strengthen the reliability of forensic science and its use in the courtroom. We are committed to improving forensic science.

That’s good to hear. But we’ve heard that before. And the Justice Department’s reaction to the P-CAST and NSFC reports was basically hostile. And again, given the history here, you’d expect perhaps some humility. Instead, we get empty promises and assurances.

Much of the rest of what Rosenstein proposes consists of additional funding, training and supplying of crime labs. That’s all well and good, but what’s really needed are assessments of crime lab methods and practices from scientists outside the field of forensics. The areas of forensics with no basis in science need to be discarded. Those that have some evidentiary value but are heavily reliant on the judgment and expertise of analysts (as opposed to more objective measures) need safeguards against cognitive bias — the unconscious bias that can creep into the work of even the most conscientious analysts. That means testing should be done blind whenever possible, and analysts’ work should be regularly, blindly and randomly double-checked for accuracy. We need to understand the incentives and be sure they’re properly structured — crime labs should never be under the auspices of police agencies or prosecutors’ offices. (Which means that, yes, the “FBI Crime Lab” shouldn’t exist.)

Rosenstein’s line here seems to be “trust us, we’ve got it right this time.” But we’ve heard that before. The deputy attorney general then closes with a reference to a movie.

Has anyone seen the movie My Cousin Vinny? In the film, a young woman takes the witness stand and surprises everyone by demonstrating her expertise about automobiles. It does not necessarily take a Ph.D. scientist to be an expert witness and to provide information that will be helpful to a jury. And it certainly does not require proof of a known error rate. Many factors may influence the weight of the evidence – how much the jury should rely on it – but we could rely on juries to make those decisions as long as the witness is competent and responsible, and the judge gives appropriate instructions about how to evaluate the testimony.

I’m not sure what this is supposed to tell us, especially given that defense attorneys and prosecutors alike complain about how fictional portrayals of forensics have created unrealistic expectations in jurors. We’ve asked judges — who are trained to think, write and analyze like lawyers — to determine the validity of scientific evidence. We’re asking them to become scientists. That’s likely why judicial opinions about scientific evidence are often so lacking. As one legal expert on science and evidence once told me, the opinions often read like a college student essay in a logic class. They start with their conclusion and work back from there. Much of the time, judges simply note how many other courts have already admitted a given form of evidence. Deferring to precedent is about the most unscientific way of assessing scientific evidence one can imagine. If science deferred to precedent, we’d still be afraid of sailing off the edge of the Earth, which we would still believe lies at the center of the universe. But that’s how the legal system works. And legal people are, naturally, in charge the legal systems’ rules. Thus, we use legal rules to assess science in the courtroom.

It also isn’t clear what argument Rosenstein is rebutting with his pop culture reference. In “My Cousin Vinnie” — which again is fiction — Marissa Tomei’s character is allowed to testify as an expert after correctly answering a series of questions about automobiles. She then offers testimony that deflates the prosecution’s case. But no one is arguing that only highly-credentialed experts should ever be permitted to testify. Credentials are a shortcut for assessing credibility, and often not a particularly accurate one. The argument here is that any expert, regardless of qualifications, shouldn’t be able to tell juries things that aren’t true, aren’t backed by science or aren’t knowable.

To illustrate the point, let’s change the plot of the movie a bit. Tomei’s character obviously knew more about cars than the average judge, lawyer or juror. But let’s say that instead of offering her bona-fide expertise about the car’s suspension, she had given some hyper-technical testimony that also happened to be completely fallacious. The problem is that only someone similarly well-versed in the inner workings of automobiles could have known her testimony was false. Sure, the prosecution could put on a mechanic to rebut her. You’re now leaving it up to the jury to decide which expert is correct. But we’ve just established that this is subject matter that’s well beyond the jury’s expertise — that’s why experts were brought in to testify in the first place. If the jury had insufficient knowledge before the trial, presenting two experts with directly contradictory testimony puts them in no better position to know good testimony from bad than they were before the trial. Again, this is where charisma, charm and the art of persuasion can become more important than expertise.

We need to keep the fallacious testimony out of court in the first place. That’s easier said than done. We know that judges have been poor gatekeepers, as have the appellate courts. Perhaps we could assemble panels of scientists to assess the validity of the existing areas of forensics and then new fields as they pop up. That presents its own challenges, most of them political, but it would seem to be significantly better than asking judges.

The one approach that seems most obviously wrong is the one that Rosenstein is suggesting — that we ignore history, incentives and common sense and simply put our trust in prosecutors, police agencies and other law enforcement officials to get it right.