The problem with modern polling, in one chart

Most political polling these days is seriously skewed. But the problems with polling aren’t oversamples of partisans bending results to one side’s preferred perspective — the problem is that Americans are increasingly changing the way they communicate.

Traditionally, pollsters have gauged voter or consumer opinions by calling their targets at home. Standard operating procedure in the industry requires pollsters to randomly select a set of survey respondents and call them repeatedly — up to six times — in an effort to get an answer. If they can’t reach those voters by the sixth try, only then will they move on to a new respondent.

But voters aren’t waiting around by their clunky old landlines anymore. In fact, the Centers for Disease Control reported that more than half of us either don’t own a landline phone or don’t use their phone as their primary means of communication. Today, we live on our cellphones.

How prevalent are cellphones? In a smart piece written for The Huffington Post last week, Democratic pollster Brian Stryker pointed to industry statistics that show there were 326 million active cellphones in the U.S. as of December 2012 — 1.04 per person, infants and children included, in the entire country.

Here’s the dramatic change in households that rely on cell phones versus landlines that Stryker used to illustrate his point:

Cellphone and landline usage, 2003-2013


Graphic courtesy Anzalone Liszt Grove Research

Those who use cellphones look remarkably different, demographically speaking, than those who use landlines. More than 60 percent of adults under age 45 use only their cellphones, Stryker reported, using data collected by the CDC, versus just 13 percent of those 65 and older. Hispanics are much more likely to rely solely on their cells than any other race.

Stryker’s polling from 2012 shows those who own only cellphones are much more likely to lean towards Democrats than those attached to landlines. Cell-only respondents leaned toward Democrats by 11 percentage points; those who answered surveys on landlines leaned toward Democrats by just 2 percentage points.

The implications are clear: If you want to produce a survey that tilts distinctly Republican, call landlines only. If you want to produce a Democratic-skewed poll, stick to cell phones. If you want a survey that accurately represents the views of the modern electorate, controlling for the percentage of cellphone-only users is just as important as making sure your sample accurately reflects gender, race and education breakdowns of the broader population.

(A minor side note: The notion that media pollsters somehow “skew” their results to fit a preferred narrative is pretty far-fetched. After all, as the political cliche goes, the only poll that matters is Election Day, and if a pollster is terribly far off the final result, credibility is at risk. In the polling world, that means risking their very careers. Sure, there are outlets that poll to push a partisan agenda — Public Policy Polling is unquestionably a Democratic firm, while Rasmussen certainly favors Republicans — but the vast majority of pollsters, even the partisans, have to get the final result correct or they risk their future employment prospects. What political candidate wants to get overly optimistic results and, in the process, calibrate campaign pitches incorrectly?)

But the trouble is reaching cellphone respondents is much more expensive than reaching voters on landlines. The Pew Research Center and others have reported that the percentage of respondents who complete entire surveys — some of which can last half an hour or longer — has fallen precipitously, from 36 percent in 1997 to just 9 percent in 2012. Lower completion rates mean pollsters have to make more calls. (Costs are even higher for automated polls; federal law prevents companies from sending automated calls to cellular phones, meaning those normally low-cost pollsters either have to shell out far more to pay actual human beings to make the calls, which seriously eats into their bottom lines, or ignore cell phones altogether, which skews their results).

We asked Scott Clement and Peyton Craighill, The Washington Post’s expert pollsters, to weigh in on how the proliferation of cell phone-only households is changing the polling industry. Peyton wrote back:

This is a vitally important issue for survey quality, and quite frankly, something that is not new for media/public pollsters at all. The fact that partisan pollsters are making hay about this now is commendable, but awfully late in the game. The proliferation of robopolls may be eating into the business model for partisan firms and they feel the need to point out the necessity for dialing cell phones. Could be some self-service going on here. Regardless of the motivations, it’s a step in the right direction.

As for our views on sampling cell phones, it’s absolutely essential to include for any reliable poll. Without reaching the cell-only population you are missing up to 40 percent of eligible poll respondents for a national poll. And that can go higher in some states and localities. The Washington Post-ABC News national poll completes 30 percent of its interviews among cellphone only respondents. We sample up to 50 percent of cellphone only for Washington, D.C. city polls. By sampling only people on landlines, you systematically exclude certain groups of people, especially younger people and non-whites. And as Brian notes in his analysis, the attitudes of the cellphone only crowd are materially different from similar landline only people.

There is no indication that the move toward cell phone only is slowing down or reversing. If anything it is speeding up.

Stryker’s conclusion: Pollsters need to start including more cellphone respondents in their surveys, post-haste. He recommends at least 15 percent of samples should be comprised of cell-only voters to accurately reflect the low-turnout 2014 midterm electorate, and an even higher percentage of the larger 2016 presidential electorate.

So when evaluating a poll, after checking the partisan breakdown, be sure to check just how much of a given survey was conducted among cellphone users. It’s another grain of salt one should use when gauging the electorate — and the accuracy of any given survey.

Reid Wilson covers state politics and policy for the Washington Post's GovBeat blog. He's a former editor in chief of The Hotline, the premier tip sheet on campaigns and elections, and he's a complete political junkie.
SECTION: {section=politics, subsection=null}!!!
INITIAL commentConfig: {includereply=true, canvas_permalink_id=washpost.com/8bvh5zpd9k, allow_comments=true, commentmaxlength=2000, includeshare=true, display_comments=true, canvas_permalink_app_instance=m6yzjj840m, moderationrequired=false, display_more=true, includefeaturenotification=true, comments_period=14, canvas_allcomments_id=washpost.com/km4ey0dajm, defaultsort=reverseChronological, includevoteofftopic=false, allow_videos=false, includesorts=true, markerdisplay=post_commenter:Post Commenter|staff:Post Writer|top_commenter:Post Forum|top_local:Washingtologist|top_sports:SuperFan|fact_checker:Fact Checker|post_recommended:Post Recommended|world_watcher:World Watcher|cultuer_connoisseur:Culture Connoisseur|weather_watcher:Capital Weather Watcher|post_contributor:Post Contributor, childrenitemsperpage=3, includeheader=true, includeverifiedcommenters=true, defaulttab=all, includerecommend=true, includereport=true, maxitemstop=2, source=washpost.com, allow_photos=false, maxitems=7, includepause=true, display_ugc_photos=false, canvas_allcomments_app_instance=6634zxcgfd, includepermalink=false}!!!

UGC FROM ARTICLE: !!!

FINAL commentConfig: {includereply=true, canvas_permalink_id=washpost.com/8bvh5zpd9k, allow_comments=true, commentmaxlength=2000, includeshare=true, display_comments=true, canvas_permalink_app_instance=m6yzjj840m, moderationrequired=false, display_more=true, includefeaturenotification=true, comments_period=14, canvas_allcomments_id=washpost.com/km4ey0dajm, defaultsort=reverseChronological, includevoteofftopic=false, allow_videos=false, includesorts=true, markerdisplay=post_commenter:Post Commenter|staff:Post Writer|top_commenter:Post Forum|top_local:Washingtologist|top_sports:SuperFan|fact_checker:Fact Checker|post_recommended:Post Recommended|world_watcher:World Watcher|cultuer_connoisseur:Culture Connoisseur|weather_watcher:Capital Weather Watcher|post_contributor:Post Contributor, childrenitemsperpage=3, includeheader=true, includeverifiedcommenters=true, defaulttab=all, includerecommend=true, includereport=true, maxitemstop=2, source=washpost.com, allow_photos=false, maxitems=7, includepause=true, display_ugc_photos=false, canvas_allcomments_app_instance=6634zxcgfd, includepermalink=false}!!
Comments
SECTION: {section=politics, subsection=null}!!!
INITIAL commentConfig: {includereply=true, canvas_permalink_id=washpost.com/8bvh5zpd9k, allow_comments=true, commentmaxlength=2000, includeshare=true, display_comments=true, canvas_permalink_app_instance=m6yzjj840m, moderationrequired=false, display_more=true, includefeaturenotification=true, comments_period=14, canvas_allcomments_id=washpost.com/km4ey0dajm, defaultsort=reverseChronological, includevoteofftopic=false, allow_videos=false, includesorts=true, markerdisplay=post_commenter:Post Commenter|staff:Post Writer|top_commenter:Post Forum|top_local:Washingtologist|top_sports:SuperFan|fact_checker:Fact Checker|post_recommended:Post Recommended|world_watcher:World Watcher|cultuer_connoisseur:Culture Connoisseur|weather_watcher:Capital Weather Watcher|post_contributor:Post Contributor, childrenitemsperpage=3, includeheader=true, includeverifiedcommenters=true, defaulttab=all, includerecommend=true, includereport=true, maxitemstop=2, source=washpost.com, allow_photos=false, maxitems=7, includepause=true, display_ugc_photos=false, canvas_allcomments_app_instance=6634zxcgfd, includepermalink=false}!!!

UGC FROM ARTICLE: !!!

FINAL commentConfig: {includereply=true, canvas_permalink_id=washpost.com/8bvh5zpd9k, allow_comments=true, commentmaxlength=2000, includeshare=true, display_comments=true, canvas_permalink_app_instance=m6yzjj840m, moderationrequired=false, display_more=true, includefeaturenotification=true, comments_period=14, canvas_allcomments_id=washpost.com/km4ey0dajm, defaultsort=reverseChronological, includevoteofftopic=false, allow_videos=false, includesorts=true, markerdisplay=post_commenter:Post Commenter|staff:Post Writer|top_commenter:Post Forum|top_local:Washingtologist|top_sports:SuperFan|fact_checker:Fact Checker|post_recommended:Post Recommended|world_watcher:World Watcher|cultuer_connoisseur:Culture Connoisseur|weather_watcher:Capital Weather Watcher|post_contributor:Post Contributor, childrenitemsperpage=3, includeheader=true, includeverifiedcommenters=true, defaulttab=all, includerecommend=true, includereport=true, maxitemstop=2, source=washpost.com, allow_photos=false, maxitems=7, includepause=true, display_ugc_photos=false, canvas_allcomments_app_instance=6634zxcgfd, includepermalink=false}!!
Show Comments
Next Story
Reid Wilson | March 11