Experts aren’t very good at predicting Supreme Court cases

June 25, 2012

I posted last week on polls of former Supreme Court clerks and constitutional scholars, all suggesting that the individual mandate was toast. For both groups, their new view is a reversal of their old view, which was that the individual mandate was legally bulletproof.

All these predictions should probably be taken with a grain of salt. Reader Nicholas Marantz passes along two studies showing that legal experts aren't much better at predicting the case outcomes than a coin toss. Sometimes, they even do worse.

The Columbia Law Review looked back at predictions made during the Supreme Court's 2002 term. Here is a chart showing how well 86 court watchers did, broken down by field:

You'll notice attorneys look to do especially well at predicting the outcome of Supreme Court cases, although the authors caution against reading too much into that, given the small number of attorneys represented in the survey. A statistical model the authors came up with — one that relied on some basic characteristics about each case — did a lot better than most, accurately predicting the outcome of 75 percent of the cases.

Washington University's Andrew Martin did an even deeper dive into the issue and looked at the types of cases that experts are good (and not so good) at predicting. They turn out to do the absolute worse on cases involving questions of "economic activity," which happens to be the crucial issue in the health reform challenge.

It's the one area where the experts predicted the decision accurately less than half the time. In other words, a coin toss would make a better prediction model. Take a look:

Martin's computer model, meanwhile, did a pretty great job of predicting the outcome of the exact same cases.

Comments
Show Comments
Most Read Business
Next Story
Suzy Khimm · June 25, 2012