Polling is under siege. Recent failures in election polling in Britain, Israel and a few high-profile states in the U.S. midterm elections have been squarely attributed to an increased unwillingness of many Americans to take surveys (what the industry terms “declining response rates”). Some in Congress have called for significant reductions in funding for the census and making data collection for the Census Bureau voluntary, despite the critical role such data plays in hundreds of government policy decisions — not to mention its importance for businesses that use such data for a range of strategic decision-making.
Yet, with apologies to Mark Twain, the simple truth is that the death of probability-based, scientifically rigorous and highly precise polling has been greatly exaggerated. Polls serve a critical role in democracy and, contrary to noisy critics, continue to provide highly accurate estimates of public sentiment.
Recent research into Pew Research Center, ABC-Washington Post and CBS-New York Times telephone polling finds little growth in inaccuracies in polls over the past 20 years, despite response rates having dropped from the 30 percent range to the single digits. Indeed, reported estimates show absolutely no increase in bias, as illustrated in the chart below. Random surveys of Americans, dialed via cellphones, attain remarkably accurate cross sections of Americans by most key demographics, such as age, race and level of education. It is easy to claim that millennials don’t use their phones for anything but texting, but when the phone rings, they answer. (Arguably, having not lived through the past decade of telemarketing land-line glut, they may even approach telephone surveys with less reticence, not more.)
Other evidence of polling’s resistance to error comes from polling aggregator Nate Silver, who examined how far off predictions have been for almost every election held in the United States since 2000. Notably, Silver did not identify any decline in the accuracy of presidential, gubernatorial, House or Senate polls.
And it makes sense. Scientifically, it does not matter whether 1 in 5 or 4 in 5 Americans hang up on pollsters, as long as those who hang up are largely similar to those who participate. While there are differences between those who participate and those who do not, nearly 20 years of research finds there are far more similarities between the two groups than differences. Democracy would probably be better served if more people chose to participate in polls, but at the same time, lack of participation is not leading to a significant effect on overall accuracy.
Accuracy just isn't newsworthy
When pollsters “miss,” it is inevitability big news. This has been true since 1948, when some pollsters predicted a Thomas Dewey win over Harry Truman. It is not exactly newsworthy to also talk about the fact that pollsters then went on a 64-year run of correct predictions, before one or two firms in 2012 predicted a close Mitt Romney win. Pollsters do not bat 1.000. But their accuracy is noteworthy, if not newsworthy.
Polling is now a field with many approaches and techniques, and some polls are better than others. Low-cost, low-quality alternatives to traditional telephone polling will inevitably miss the mark. But even high-quality polls will sometimes be wrong, particularly in an age of deep partisan divide, when many elections are characterized by one- or two-point victories, not 10-point landslides.
It has been said that political polling is uniquely challenged in that the population being surveyed does not yet exist at the time of the survey: It is surveying a future population of those who actually show up at the polls on Election Day. Pollsters have to rely on statistical modeling to fill the gap, and because who shows up has been more variable of late, there is much more pressure to get the models right. It must be noted that these models have worked tremendously well for decades, and recent advances offer the promise of maintaining past performance, at the very least.
Polling has faced challenges in every decade of the modern age, and the industry has met and successfully overcome each of them. Polling is not infallible, and no one knows what the future will bring. There will surely be future misses, but polls continue to be the most trustworthy and reliable opportunity for people to have a voice.
If you get called, choose the right to be heard over your right to refuse. Do not undermine the role of polling in democracy with unfounded claims of its demise.
David Dutwin is executive vice president and chief methodologist for SSRS, a survey research firm outside Philadelphia. He is also a senior fellow at the Program for Opinion Research and Election Studies at the University of Pennsylvania and serves on the Executive Council of the American Association for Public Opinion Research. Opinions expressed here are solely his own.