The Washington PostDemocracy Dies in Darkness

The 2017 elections boosted confidence of exit pollsters — and their future competitors

A voter casts a ballot at San Francisco City Hall on Nov. 8, 2016. (David Paul Morris/Bloomberg News)
Placeholder while article actions load

As Democrats swept statewide elections in New Jersey, Virginia and Alabama this year, the precursor of a very different competition was playing out across the nation’s newsrooms, with significant consequences for interpretation of election results going forward.

The ubiquitous Election Day exit poll was conducted by a consortium of ABC News, CBS News, CNN and NBC News, known as the National Election Pool. It's long been the quickest and most commonly cited source for how different demographic and political groups voted since CBS News first conducted a national exit poll in 1972. (The Washington Post subscribed to receive NEP exit poll results in Alabama and Virginia, and has been a subscriber in previous years.)

But two longtime NEP members — Fox News and the Associated Press — left the alliance this year and conducted experimental surveys, looking to fill the role of exit polls in the 2018 and 2020 elections. Fox and the AP are both taking a different approach to surveying voters, instead using a combination of Internet- and telephone-based samples, with interviewing beginning in the days leading up to Election Day.

[Fox News is trying to reinvent the exit poll. The survey strategy involves people who don’t vote.]

In interviews over the past week, both the NEP and Fox expressed satisfaction with the performance of their voter polls in statewide elections this year. The AP will be completing its analysis of results in the coming months and says it will be made public as it has with previous experiments testing alternatives to the traditional exit poll.

The departures of Fox and AP raised questions about how long the NEP will continue, given its extraordinary expense, its reduced clientele and criticism over its preliminary results in 2016 battleground states. The 2016 errors were cited by Fox News this spring as a reason for exiting the partnership. Politico, which first reported the AP’s exit from the consortium, described the NEP poll as “on life support.”

To the contrary, the NEP and its longtime research contractor, Edison Research, say they are planning to continue conducting exit polls with the smaller group of four sponsors, and they are largely satisfied with its performance. “We have a commitment from our clients through 2020,” Joe Lenski, executive vice president of Edison, said in a phone interview Monday.

In Alabama, Lenski said, results from the first two of three waves of interviews that networks began reporting at 5 p.m. found Doug Jones and Roy Moore tied, close to Jones’s eventual winning margin of 1.5 percentage points. In Virginia, the first wave of reportable exit polling found Ralph Northam leading by five points over Ed Gillespie, with Northam winning the race by a nine-point margin.

[Alabama election exit poll results]

The preliminary vote estimates are not reported by the consortium on election nights in an effort to avoid suggesting a likely outcome, but they are used in planning election night coverage and providing a sense of the overall direction of the race. The exit poll is weighted to actual vote returns as they are counted.

“We feel pretty good about giving our clients and subscribers the right impression,” said Lenski in a phone interview Monday. In a statement Friday, the NEP described the Alabama exit poll as “extremely accurate.” “We believe the exit polls remain the statement of record for election coverage, providing insights into who turned out and how voters made their decisions — as only a poll that goes and talks directly to actual voters can do.”

NEP was put on the defensive by the 2016 election, but Lenski defended the network exit poll’s record that year. He noted that “the national poll was right on at 5 o’clock showing a three-point Clinton lead in the national popular vote, and the election ended up being just over two points,” referring to preliminary results based on two of the three waves of exit polls. That result was similar to the overall accurate performance of national pre-election polls.

On state-level surveys, Lenski noted that in preliminary exit poll results available at 5 p.m., Clinton had a one- to five-point advantage in five states that Trump eventually won. At poll closing time that evening, exit polls had Clinton ahead by one to four points in four states that she eventually lost. 

“I think those of us that use exit polls know that a one- or two-point lead in an exit poll does not mean that you can draw much more conclusion. It’s a 50-50 chance either candidate can win,” Lenski said. “None of those margins are anywhere near callable based upon an exit poll survey as evidenced of the fact that none of our clients or subscribers made any projections in those states based on exit poll estimates.”

Still, the AP and Fox sought a separate course.

Fox News pollsters gained confidence in testing a replacement for exit polls this year, according to Arnon Mishkin, who directs Fox’s Decision Desk. “Polls sort of nailed it in Virginia and New Jersey. In Alabama, it gave us a good perspective on the election” he said in an phone interview last week. The Alabama poll “said it would be somewhere between a healthy Jones victory and a chance Moore ekes it out.”

“One of the reasons we did this experiment is that almost half of Americans don’t vote on Election Day — this is particularly true in the West.” said Mishkin. The network exit poll typically includes an additional phone survey in states in which a large share of voters are expected to cast ballots early. “We think it’s smart economics to have one instrument,” said Mishkin about Fox’s approach.

The AP was motivated to experiment by cost as well as the increasing share of the electorate who vote early, according to David Scott, deputy managing editor for operations. “America votes differently. In Nixon’s election, 95 percent voted in person. In 2012, 60 percent voted in person” Scott said in a phone interview last week. The United States Election Project reported 47 million ballots were cast early in the 2016 election, citing reports from jurisdictions, accounting for roughly 34 percent of all ballots.

“We’re trying to devise a methodology that aligns with how Americans are voting today” Scott said, “that is our overriding concern. That’s not a criticism of the exit poll as it has existed. We’ve always tried to contribute to the design of that poll, make it better, it’s why we are continuing to experiment.”

Challenges going forward

All three efforts face significant methodological challenges in accurately gauging voter opinions in the short time frame demanded by Election Day deadlines. Unlike interviews with voters as they exit polling stations, Fox and the AP cannot be certain their respondents will actually cast a ballot, an action that people tend to over-report. Instead, they will need to rely on self-reports of voting, as well as likely voter models, to differentiate and non-likely voters in the same way as pre-election polls.  

Both newcomers will also need to wrestle with effectively combining data from a statistical “probability” phone sample drawn from voter lists for Fox and Random Digit Dialing by AP, with an online “non-probability” sample. The AP's Scott says that in this year’s experiments, a smaller phone sample was used to calibrate a large online survey.

As with all pre-election polls, the Fox and AP surveys and the NEP exit poll must deal with the potential that one candidates’ supporters are more likely to take the survey, which can lead to nonresponse bias.

Lenski said the exit poll has weighted to correct for differing response rates by sex, age and race for decades based on interviewer observations. “In Alabama, African Americans were more likely to respond to the exit poll than white voters, so we had to adjust for that as well,” Lenski said. But Edison is also exploring ways to better correct for greater participation among college graduates. “We’re in a time politically where that is the difference between how college-educated voters are voting and non-college-educated voters are voting has become more critical than ever.”

As all three groups sort out those challenges, one clear consequence of the Fox and AP efforts is a larger number of data sources for reporting the makeup of the electorate and how different groups voted, which Fox’s Mishkin sees as a clear positive. “I think the world is better served by having two X-rays rather than one exit poll.”