A pair of political science professors are out with a seemingly significant study: Despite Hillary Clinton saying she would be president if not for James B. Comey — and FiveThirtyEight, among others, lending credence to that claim — “We don’t think so,” declare Costas Panagopoulos and Aaron Weinschenk, who wrote up their study for the Monkey Cage.
It would seem a pretty big counterpoint to a popular bit of emerging conventional wisdom on the left. But in reality, it’s not all that contradictory. Their conclusions may be different in tone, but their research largely confirms that Comey may have indeed tipped the scales — with the key phrase there being “may have.”
While FiveThirtyEight and others have said it’s likely that Comey did, Panagopoulos and Weinschenk were basically unable to prove that he did. They were looking for a statistically significant impact late in the 2016 election, and they didn’t find it. But by that standard, basically nothing in the final month-plus of the campaign truly mattered, because nothing moved the needle that much.
The study attempts to balance a number of factors that are traditionally believed to have helped either Clinton or President Trump. In the end, it found that only three things affected the race in a statistically significant way: President Obama’s popularity, the Democratic National Convention and the first presidential debate. And all of those accrued to Clinton’s benefit:
We measure the effect of the Comey letter by including a variable to examine changes in the polls on Oct. 28 — the day the letter was released — and in the days afterward. If the letter hurt Clinton, we should find a statistically significant decline in her lead over Trump, above and beyond what the other variables would tell us. This is a standard way of measuring whether a campaign event has a durable effect on the polls.But that’s not what we find. Our model shows that the Comey letter led to a small drop in Clinton’s lead, but that shift is not statistically significant. In other words, there is no clear evidence that the letter cost Clinton votes. When we specify our model in different ways — for example, looking at the impact for just the three days after its release — we again do not find evidence that the letter moved the polls.
So at once, the study suggests Comey’s letter “led to a small drop in Clinton's lead” but that it did not “move the polls” in a statistically significant way. But that’s not really all that far off what FiveThirtyEight's Nate Silver said in his piece arguing that Comey likely cost Clinton the race:
Clinton’s standing in the polls fell sharply. She’d led Trump by 5.9 percentage points in FiveThirtyEight’s popular vote projection at 12:01 a.m. on Oct. 28. A week later — after polls had time to fully reflect the letter — her lead had declined to 2.9 percentage points. That is to say, there was a shift of about 3 percentage points against Clinton. And it was an especially pernicious shift for Clinton because (at least according to the FiveThirtyEight model) Clinton was underperforming in swing states as compared to the country overall. In the average swing state, Clinton’s lead declined from 4.5 percentage points at the start of Oct. 28 to just 1.7 percentage points on Nov. 4. If the polls were off even slightly, Trump could be headed to the White House.But it’s not credible to claim that the Comey letter had no effect at all.
This new study was effectively trying to control for all those other possible reasons. There is a lot going on in the final week of a campaign, so isolating one factor as the only reason for a shift over several days is difficult — which Silver admits.
That’s why the new study is valuable. But its authors also set a high bar for saying Comey’s letter actually, definitively mattered. Let’s remember that Trump won by less than one percentage point in the three decisive states: Michigan (0.2 percent), Pennsylvania (0.7 percent) and Wisconsin (0.8 percent). If Comey’s letter shifted the race by even one point across the board nationally, then, it will have changed the result. From a political science perspective, though, you can’t call a one-point shift significant, because it could simply be statistical noise.
And as I mentioned at the top, this standard suggests basically nothing truly mattered: not the second and third presidential debates, not a late announcement of a 25 percent rise in Obamacare premiums, and not even the “Access Hollywood” tape. Basically all that mattered, according to this standard, were Obama’s approval rating and things that happened Sept. 26 (the date of the first debate) or earlier. Nothing else registered a significant shift. The candidates might as well have ended their campaigns after that first debate and not campaigned for the final six weeks.
This is a little unfair, but it’s the logical extension of requiring statistically significant proof. It’s possible that we are just such a polarized country that it’s difficult for any one thing to shift the polls in a statistically significant way in the final days or even weeks of a campaign. That doesn’t mean those things didn’t actually change things, but those shifts are too small to declare the events significant. And 2016 only needed a very small X-factor to change the result.
Silver believes there is a credible case to be made that the three-point shift in the week after Comey’s letter was in large part due to the letter, which he notes got a huge amount of news coverage. The new study suggests this may not be the case, which Silver readily conceded was possible. They are basically saying the same thing; the new study just sets a very difficult standard of evidence and seems to be written as a response to a version of the Comey conventional wisdom that doesn’t quite exist.