In the wake of last year’s presidential election, the polling and political projection industries came in for a beating, accused of missing one of the biggest political stories in a generation. A new report examining what happened comes to a split conclusion: National surveys were generally accurate in projecting the popular vote but state polls had “a historically bad year” in forecasting the results in the electoral college.
Donald Trump won the presidency by rolling up 306 electoral votes to Hillary Clinton’s 232. But Clinton won the popular vote by almost 3 million votes and a margin of 2.1 percentage points. This was the second time in the past five elections in which a Democrat won the popular vote and a Republican won the electoral college.
The results produced a deluge of criticism about campaign coverage, given near-universal predictions of a Clinton victory, and why Trump was so underestimated heading up to Election Day. The authors of the report summed up the public response to the outcome this way:
“The day after the election, there was a palpable mix of surprise and outrage directed toward the polling community, as many felt that the industry had seriously misled the country about who would win.”
The criticism prompted the American Association for Public Opinion Research (AAPOR) to convene a 13-member committee to study in detail to what extent the polls were off and why. The panel’s report was released Thursday.
Earlier this week, Clinton blamed her loss in part on the decision by FBI Director James B. Comey to send a letter to Congress on Oct. 28 announcing that investigators had come across a fresh trove of her emails and that the investigation of her use of a private email server as secretary of state was being reopened.
“If the election had been held on October 27, I would be your president,” Clinton said in a public interview with CNN’s Christiane Amanpour.
The AAPOR report sought to answer the question of how much Comey’s letter contributed to Trump’s victory but stopped short of providing a conclusive answer. “The evidence for a meaningful effect on the election from the FBI letter is mixed at best,” the report states.
On the eve of the election, various indicators pointed to a Clinton victory, according to the report. Among them were the patterns of early voting in swing states, forecasters’ predictions and polling data from the Upper Midwest, which “showed Clinton leading, if narrowly, in Pennsylvania, Michigan and Wisconsin.” Ultimately Trump carried those three states by just 77,744 votes, or barely one-half of one percentage point.
The report’s two main findings about polling underscore that it was a treacherous year for pollsters, and therefore predictions. The AAPOR committee concludes that the national polls “were generally correct and accurate by historical standards,” and that they were more accurate than in 2012. The polls, on average, pointed to a Clinton victory in the popular vote by about three percentage points. Her eventual advantage was well within the margin of error of the national polls.
But where elections are decided, in state-by-state contests, things were not so rosy for pollsters. State polls were historically bad — the report calls it the largest error in state polling of elections starting in 2000 — and the key failure was the underestimation of Trump’s support. This was particularly true in the Upper Midwest, where the election was decided.
The AAPOR team also found flaws among those organizations that produce poll aggregates and projections of results and pointed to these predictions as one reason so many people were surprised by the outcome of the election. “They helped crystallize the erroneous belief that Clinton was a shoo-in for president,” the report states.
The report sought to draw a distinction between polls, which the authors call “a snapshot in time,” and prediction models. But there was an explicit warning about the limits of all such measurements. “Caution and humility seem to be in order for pollsters and those who use polls,” the report states.
The authors sought to determine why Trump’s vote was underestimated, particularly in state surveys. One reason, they concluded, was that many voters waited to the end to decide how to vote, or changed their minds in the final days. These errors were “substantial and problematic in several consequential states,” the report concludes.
Among the states where it mattered most was Wisconsin, a state Republicans had not won since 1984 but which Trump carried by less than a percentage point. But there were also problems in several other states that proved crucial, including Michigan and Pennsylvania, two other states that had voted for Democratic nominees in six consecutive elections.
Exit polls found that late-deciding voters backed Trump by substantial margins in Florida, Michigan, Pennsylvania and Wisconsin in what broke the back of the Clinton campaign’s path to victory. The report notes: “This can be seen as good news for the polling industry. It suggests that many polls were probably fairly accurate at the time they were conducted.”
That might be small consolation to the polling community, given public perceptions that forecasts in general were off the mark in 2016.
One clear weakness in state polls, according to the report, was the failure of many pollsters to weight their data on the basis of educational achievement. In 2016, there was a significant education gap in voters’ behavior, with non-college-educated voters supporting Trump heavily in comparison to those with college degrees.
Many state polls continued to base their results on samples that included too many college-educated voters and too few without degrees, despite the fact that the education gap was well documented early in the campaign.
Another possible cause for errors in state polls was the change in the composition of the electorate and of turnout by different groups of voters compared to 2012. The report suggests this was a problem but the authors said they lacked some additional data from the federal government that would allow for more detailed study.
A much smaller factor cited as a possible reason polls in the Midwest tended to underestimate support for Trump was that he appeared as the first name on ballots in Florida, Michigan and Wisconsin.
One theory often advanced is that many people who supported Trump were unwilling to tell pollsters that they favored him. Various tests of this theory “yielded no evidence to support it,” the report says.
The report sought to debunk the argument that the polls have a partisan bias toward Democrats. Although the 2016 surveys tended to underestimate Trump’s support, polls in 2012 and 2000 tended to underestimate the support for the Democratic presidential nominees.