Labor Day has passed, which in an election year means Americans are about to become awash in polls tapping the mood of voters as they decide which candidate to support in 435 congressional districts and dozens of U.S. Senate and governor’s elections on Nov. 6.
The midterm elections also mark the first major test of the accuracy of pre-election surveys since 2016, when polls were panned for showing Donald Trump trailing in the final days and some election forecasters predicted Hillary Clinton had a greater than 90 percent chance of a victory.
National surveys turned out to be quite accurate in gauging Clinton’s 2.1 percentage-point advantage in the national popular vote, but polls in key battleground states systematically underestimated Trump’s support, according to a 2017 report by the American Association for Public Opinion Research.
Recent surprise outcomes in Democratic primaries in New York, Florida and Massachusetts have revived questions about polling accuracy, although they also underscore a truism about nonpresidential year polling: Few high-quality polls are conducted in races that are not expected to be competitive, often leaving election watchers with a blind spot.
What lessons can election watchers take from 2016 and past midterm elections to be savvy consumers of election polls this year? Elections and polling experts contacted by The Washington Post this week pointed to several unique challenges that surveys face this fall, and they offered strategies to learn the most from pre-election surveys this year — as well as what types of results to take with a grain of salt (or a whole salt shaker).
1. The most direct polling for Congress may also be some of the least accurate
The principal contest in November is over which party will win a majority of the 435 congressional districts to control the House, with 60 to 70 districts expected to be competitive. While national polls provide highly predictive indicators such as presidential job approval and the generic ballot — whether voters want an unnamed Republican or Democrat to win — these are indirect measures by their nature. Surveys of vote preferences at the congressional districts are relatively rare in comparison, making the picture of such contests fuzzier than in presidential years.
In addition, the data journalism website FiveThirtyEight found that polling error in congressional district surveys has averaged 6.2 percentage points on the vote margin over time, higher than 5.4 points for state-level election surveys and four points for national polls.
“Pre-election polling is like a lot of other consumer products — you get what you pay for,” Michael Traugott, a professor of political science and communication at the University of Michigan, wrote in an email Thursday. “Most published polls are sponsored by news organizations, and local newspapers and television stations are facing severe economic pressures.”
A simple takeaway is to be mindful that national and local polls have strengths and weaknesses; expect larger errors in local-level polling, while national polls are less direct indicators of what’s happening in competitive districts across the country.
2. Read beyond who’s leading in the latest poll
Poll readers tend to get hung up on numbers, ignoring that they are more fluid than they might appear. While pinpoint accuracy is an unrealistic expectation, experts encouraged focusing more on what surveys say about the broader mood of voters.
“Read the midterm polls to learn about the public’s mood and concerns but don’t read them thinking you can divine who will win if you’re looking at a close election,” wrote Courtney Kennedy, director of survey research at the Pew Research Center. “Polls work, but they are just not designed to provide the kind of precision that can call a 50-50 election.”
Republican pollster Whit Ayres recommended focusing on President Trump’s approval rating rather than oft-cited “generic ballot” polls of congressional support.
“Presidential job approval, which is more stable than the generic ballot, is a better predictor in a midterm election that will be a referendum on the President” Ayres wrote in an email Thursday. “With only one exception, when the president’s job approval was 45% or below,” Ayres wrote, “his party lost a sufficient number of seats to flip control of the House in 2018,” referring to the 23 seats Democrats need to win control of the House this year.
Most national polls of registered voters currently show Trump below that mark, standing at about 41 percent nationally. In 2014, President Barack Obama’s low approval ratings were also a clear drag on Democratic candidates for U.S. Senate, with no candidate outpacing Obama’s approval mark in exit polls by more than nine percentage points.
3. Determining “likely voters” may be more difficult for polls this year
Before Labor Day, most surveys focus on results among registered voters overall, but as Election Day gets closer more poll results will focus on a subset of voters considered most likely to turn out: likely voters. Identifying likely voters can be more difficult in midterm elections when less than half of eligible voters cast ballots, and “the task is probably more difficult this year than in previous midterms,” according to Andrew Baumann of the Democratic firm Global Strategy Group.
“Usually, pollsters have looked at previous similar elections and assume turnout rates will be similar,” Baumann said, but he noted that the Democratic enthusiasm demonstrated in recent governors’ races in Virginia and New Jersey appear different from before recent midterms where Republicans had a clear turnout advantage. Finding a way to effectively balance voters’ past turnout habits and current enthusiasm about voting will be critical, Baumann said.
Republican pollster Kristin Soltis Anderson of Echelon Insights offered some similar guidance. “If a pollster is relying mostly on past vote history to judge who is a ‘likely voter’ they should be sure they are including a way for ‘newly likely voters’ — those who are unusually energized or tuned in to this midterm compared to previous midterms — to make their way into the survey.”
One way poll watchers can keep an eye on how turnout may impact the election is by paying attention to measures of enthusiasm for voting. If polls consistently show Democrats matching Republican certainty to vote, it’s a sign that Republicans are unlikely to have the same turnout advantage as they had in 2014 and 2010.
4. Polls are “snapshots,” and the margin of error works differently in pre-election polls
Most readers are familiar with the margin of sampling error commonly reported in surveys, but it’s often misinterpreted in pre-election polls.
As Baumann explained, “If a poll has a margin of error of plus or minus four points and the horse race is at 48 percent for the Democrat and 43 percent for the Republican,” the margin of error applies to each candidate’s estimate of support. “It means that there is a 95 percent chance that the Democrat is between 52 and 44 percent and a 95 percent chance the Republican is between 47 and 39 percent.”
In practical terms, this means that the a candidate’s advantage needs to be between 1.5 to two times the margin of error for one to be 95 percent certain a candidate actually has a lead. That doesn’t mean a candidate doesn’t actually have a lead if the result falls within the range of sampling error, only that the survey is not precise enough to detect it with high confidence.
Put simply: If a poll says a candidate is “leading” by a small margin, that lead is unlikely to be statistically significant.
Kennedy of Pew Research also cautioned against reading too much into election forecasts that are reported with decimal-point accuracy. This “implies a false sense of precision about how closely those forecasts are likely to line up with the actual outcome,” she said. Baumann of the Democratic firm Global Strategy Group warned polls against reporting results with decimal places for the same reason.
A third caution experts cited is that polls conducted weeks before Election Day may fail to capture late swings among undecided voters, which can lead to appearing less accurate when results come in. This was a factor in 2016, in which Trump won late-deciding voters in a slew of battleground states.
“The first thing for poll watchers to keep in mind is that polls are a snapshot of a race, not a direct prediction of the results on Election Day,” Doug Schwartz, director of the Quinnipiac University Poll, said in an email Thursday. “This is why pollsters are different from forecasters like 538, who are actually looking to predict the outcome of the election based on current and historical data.”
Emily Guskin contributed to this report.