What’s right with political polling (A response to Nate Silver)

August 27

Fivethirtyeight founder Nate Silver posed a provocative question on Monday: Is the polling industry in stasis or in crisis?

In a nutshell, Silver argues that polls today are largely accurate in general elections, but that “dismal” response rates are making polls increasingly reliant on demographic adjustments to make samples representative. If demographics become less predictive of how people will vote -- i.e. if African Americans stop voting 90 percent Democratic -- polls might become much less accurate in forecasting election outcomes.

First, let’s take a look at the data. Silver uses a database of polls collected by Fivethirtyeight to chart the accuracy of senate, gubernatorial and presidential pre-election polls by comparing the closeness of  survey results in the last 21 days of a campaign to the actual outcome.


The charts feature a counterintuitive finding: Over the period from 1998 and 2012 -- as poll response rates have declined dramatically and many non-standard methodologies such as automated and Internet-sampled surveys have sprung up -- poll election accuracy has not declined. Analyses by the National Council on Public Polls -- a consortium of national pollsters -- have also found that most final pre-election polls from 2000 to 2012 have been quite accurate, historically speaking.

Primary polls are a different story, according to Silver’s data, with errors on the rise in recent years.


Silver writes:

In the general election, you can model someone’s vote quite accurately by knowing a few things about him, such as his race, age, place of residence and education status. Demographic weighting can cover for a lot of problems with low and unrepresentative response rates.

Silver speculates that surveys in primaries may “be a better reflection of polls in their naked form, before demographic weighting is applied” As Silver points out, it's not all that difficult for a poll to correctly predict a general election result where a Democrat takes on a Republican. If you have a good idea of the partisanship of your sample, you will invariably come pretty close in a two-party contest. If the demographics of political support change dramatically, Silver argues, polls could become much worse forecasts of election outcomes.

Silver's worries mirror those of other pollsters, media types and academics who have pointed to declining response rates as a harbinger of the end times for traditional polling methods (i.e., surveying people through the practice of drawing a random sample of telephone numbers). The most commonly cited evidence of this threat is a 2012 Pew Research Center report tracking the organization's declining response rates over 15 years in its own telephone polls. Response rates dropped from 36 percent in 1997 to 9 percent in 2012.


National Journal’s Steven Shepard charted those data quite nicely.

So, yes, the polling industry is in a time of change. But, there's more reason for optimism than you might think. Here's why.

For one, response rates are just one reason -- and perhaps not even the most important one -- for why some poll rely on heavy statistical adjustments (aka "weighting") to fix unrepresentative samples.

If a survey’s initial sample is drawn from an unrepresentative or incomplete pool of respondents, significant adjustments and assumptions will be necessary to bring a poll’s demographics back into line. For years automated surveys relied entirely on calling landline phones, even as the cell phone-only population ballooned. While some auto-dialers now include these respondents in surveys, they often come from Internet panel samples rather than any rigorous sampling strategy. Internet panel-based surveys start with an even smaller pool of potential respondents and attempt to achieve demographic representation from the get-go by selecting respondents to match the public.

What's often ignored amid the response rate panic is that lower participation rates have generally proven to be poor predictors of survey errors in random sample polls. In the same Pew studies cited above, the organization compared results from a standard poll with a relatively low response rate to one with a far higher response rate. The results show that, after demographic weighting, the two polls produce strikingly similar results, largely mirroring results from even higher response rate government studies.

In other studies examining un-weighted survey data, lower response rates did not greatly diminish demographic representation. A study assessing more than 100 media surveys from 1996 to 2005 by researchers from three universities concluded lower response rates do in fact decrease demographic represenativeness, though not by much, challenging the assumption that response rates are a key indicator of survey data quality. Another large examination by former Census director Robert Groves found that while non-response bias clearly occurs in surveys, "there is no strong relationship between a survey's non-response rate and the non-response biases of its diverse estimates."

That said, there are clear areas of problems in response bias. Pew found its poll respondents were far more likely to volunteer and contact elected officials. In political polls, some evidence points to non-response bias after high-profile events, finding Democratic web survey takers were less likely to participate following President Obama's lackluster performance in the first debate against Mitt Romney in 2012. (No one to dwell on bad news, it seems.) But as Groves wrote, "there is no simple relationship between non-response rates and non-response biases." Lower response rates increase the risk of bias, but in practice lower response rates do not consistently produce less accurate results.

Why might this be? Two dynamics  have to operate concurrently in order for response bias to take place: 1) certain types of people must be systematically more likely to answer a pollster's call, and 2) those groups must have differing attitudes about politics or the candidates the poll is asking about. If non-response is random or unrelated to what the poll is measuring, non-response bias does not occur.

Our own Washington Post-ABC News polling provides an example of the impact of weighting data by demographics. In our June poll (which had a 23 percent response rate), President Obama’s approval-disapproval rating was 46-51 with weighted data and 45-53 with un-weighted data. The share of self-identified Democrats, Republicans and independents shifted one to two points between weighted and un-weighted data.

Another example comes from our 2012 election tracking poll results. Our final estimate of 50 percent for Obama to 47 percent for Mitt Romney was quite close to the final vote margin (51 to 47 percent). Our poll’s weights improved the poll's Obama estimate by .6 percentage points and the Romney estimate by 1.2 percentage points. In this case, weighting data to known demographic parameters brought the results closer to the true opinion of the population, but not by all that much.

The takeaway is that political surveys drawing probability samples of landline and cellular phones do not consistently see big shifts in results because of demographic weighting that may correct for differential response rates. Far from "stasis," major polling innovations on how to properly include cellular phones in the past decade has helped such surveys maintain accuracy. This is good news for election watchers, and one that helps explain the paradox of declining response rates but stable election poll accuracy.

But many election polls suffer from other major deficiencies in addition to low response rates, requiring heavy weighting or modeling of the electorate.

Silver rightly points to primary polling as an especially problematic area for pollsters, arguing that poor demographic predictors of vote choice make it difficult for surveys to correctly weight a biased sample. But there are reasons other than non-response bias to expect primary polls to be less accurate. A poll could survey the wrong population (i.e. past primary voters only), fail to accurately identify those who are actually likely to vote, interview a small sample of respondents (quite common in primary polls), or miss a late shift in voter attitudes. All of these could contribute to larger polling errors in primary races, and these are just as concerning as non-response.

So why are we optimistic about the polling industry? Quite simply, opinion polls based on probability samples continues to produce good quality survey estimates.

Show Comments
Most Read Politics
Next Story
Chris Cillizza · August 27