UPDATE: This post was originally published in late December. I'm reposting it in light of President Donald Trump's tweets this morning about polling being wrong.
Donald Trump’s poll and forecast-defying victory Tuesday immediately raised criticisms of a crisis in public opinion polling, with CNN anchor Jake Tapper saying the result would put the polling and voter projection industry “out of business.”
Election pollsters have already been under scrutiny after high-profile polling mishaps around the world, from the close Brexit vote earlier this year to the defeat Colombia's referendum making peace with FARC rebels. In the United States, Hillary Clinton's surprise victory in the 2008 Democratic New Hampshire presidential primary prompted a comprehensive report on potential causes of the errors in by the American Association for Public Opinion Research.
Survey researchers expressed varying levels of concern Wednesday about the significance of polling inaccuracies and theories on the causes. “The final high quality scientific RDD landline/cellphone national polls consistently overestimated Clinton’s share of the vote by 3 or 4 percentage points,” Jon Krosnick said in an email. Krosnick is a professor of political science at Stanford University and a widely respected expert on survey methods. “That’s a systematic error but not huge.” (Update: Final vote counts show the average of national polls overestimated Clinton’s share of the two-party vote by about one percentage point.)
Krosnick said he was not surprised by the inaccuracy of state polls, reasoning that “most state polls are not scientific — either they involve volunteer respondents instead of randomly sampled respondents, or they involve automated calling to landlines only, omitted cellphone only people.”
Monmouth University pollster Patrick Murray expressed greater concern about the way polls missed Trump’s support. “There’s a significant anti-establishment mood that polls didn’t catch — it caught some of it but not all of it,” Murray said in an interview Wednesday. “We might come to a conclusion that polls lose their precise predictive power and are best as general gauges of the mood of the electorate instead of predicting electoral outcomes.”
The causes of polling errors are typically hazy in the immediate wake of election results, due in no small part to votes continuing to be tallied well after Election Day. But even with incomplete data, it’s worth assessing how much polls differed from results and why they led to a surprising result.
An analysis of 145 polls nationally and in 16 states completed within one week of the election shows a number of interesting results. The magnitude of national and state survey errors was not far from historical levels; the more troublesome dynamic was that errors systematically overestimated Clinton’s vote margin against Trump, leading to parallel errors that did not catch a number of key states moving into Trump’s column.
Clinton won the national popular vote by two percentage points according to certified vote tallies compiled by David Wasserman of the Cook Political Report. Most individual surveys found Clinton holding a small single-digit edge over Trump, averaging to a three-point margin. Looking across individual national polls, the average difference from the final Clinton-Trump vote margin is 2.2 percentage points, much smaller than the level of error apparent when they were compared to preliminary vote results (3.4 points).
The National Council on Public Polls (NCPP for short) has analyzed the accuracy of national surveys dating back to the 1930s, using a metric called “candidate error,” which is the difference between the winning candidate’s margin over the losing candidate minus a poll’s margin, then divided by two.
The average candidate error in national polls for 2016 is 1.1, slightly lower than in 2012 (1.5) and just slightly higher than 2008 and 2004 (0.9 each) The overall size of errors this year is just below the average since 1992 (1.3) and about half the all-time average of 2.2. National poll error in the infamous 1948 election polls was nearly five times as large as 2016 (5 points).
The current estimate of candidate error in national polls is 1.7 (half of 3.4 above), slightly higher than 2012 and clearly larger than 2008 (0.9). The overall size of errors this year is a bit higher than the 1.3 average since the 1990s but below the all-time average of 2.2 and spikes in the early years, including a 5-point error in the infamous 1948 election.
One important caveat to this is that there are multiple different measures of polling accuracy — some focus on absolute estimates in support while others focus on the margin between candidates — and trends among these measures can yield varying conclusions. A different measure of national polling errors based on the difference in candidate’s estimated support and their vote share finds an average difference of between 2.9 and 3.3, with smaller errors for results that left third party candidates out.
Surveys across 17 states, including key battlegrounds, found a higher error of 1.7 using the candidate error metric, which is similar to the level of error found in all state-level surveys in 2004 (1.7), 2008 (1.8) and 2012 (1.9).
While the absolute size of most errors was far from unusual, the state-by-state results show how polling systematically showed a better picture for Clinton before Election Day. The largest consequential swing from pre-election polls to results was in Wisconsin, where Clinton averaged a six-point edge in the final week but lost the state, trailing by one-point in current tallies.
Trump won in Michigan, Pennsylvania and Florida despite polls showing Clinton ahead by narrow margins (especially in Florida). He won by a four-point margin in North Carolina, which appeared even in final pre-election polls, and far exceeded his slight advantages in Ohio and Iowa pre-election polls, winning both states comfortably.
Looking across all these states, Trump’s polling margin was three percentage points better than pre-election polls estimated. What’s striking is that many of these surveys did not miss by much — a majority came within four percentage points of the final vote margin. Yet while 52 showed Clinton faring at least four points better than the final margin, only three surveys showed Trump outperforming his final margin by that much.
Many of these two-plus differences are small enough to fall within the margin of sampling error, though taken in the aggregate they show a consistent bias. A similar dynamic occurred four years ago, with surveys underestimating Barack Obama’s margin of victory.
If only sampling error were at play, errors should be evenly scattered around population target. Assumptions of sampling error only apply to probability samples in the mix, of course, and the pattern of results suggests other types of error are at play.
Krosnick, of Stanford, suggested the answer may lie in how pollsters select likely voters and handle respondents who are less certain to vote.
“I hope those pollsters enlist teams of experts to work with them to try to figure out what happened,” said Krosnick, suggesting such teams stitch together large numbers to make estimates of small geographic areas that can be compared to election returns. “But regardless, that result in itself is not a reason to claim pollsters face a crisis.”
Both Krosnick and Monmouth’s Murray suggested exploring potential biases due to survey refusals and social desirability, such as respondents concealing their true vote preference to avoid potentially offending an interviewer. Research during the election season found mixed evidence for this, but there have been limited public examinations during the campaign.
“There might be a certain segment of the population that was not going to talk to a pollster,” Murray said. “They were going to prove us all wrong.”
Emily Guskin contributed to this report.