The Gallup Poll’s misfire in the 2012 election was caused by a variety of defects in the way the firm conducts surveys, according to the organization’s top pollster, who provided the most detailed explanation to date of how the firm plans to improve their polling accuracy in future elections.
Flanked by survey experts from academia, Gallup President Frank Newport walked through four major factors that led the poll to veer toward the “inaccurate end of the spectrum.” (Gallup’s final pre-election poll result – 49 percent Mitt Romney, 48 percent President Obama – differed not only from other surveys but also from Obama’s four-point victory.) Those factors ranged from problems with the organization’s likely voter model to unpublished landline numbers. (Much more on that below.)
The unusual gathering by the powers-that-be at Gallup — and the report issued — is a recognition of the damage done to the poll’s brand in the last election. Gallup’s polling drew intense criticism in the wake of the election and its partnership with USA Today was dissolved. (The organizations described the breakup as mutual).
Newport outlined four main reasons for the poll’s inaccuracy. “None of these factors are large in and of themselves — but they are significant enough that they made a difference in who would win the presidential election,” he added.
The four factors he listed:
1. Likely voter model shifted too far toward Romney
Gallup’s seven-question model to determine likely voters is famous in the polling world, but may have contributed to errors in 2012. While most likely voter models improved Romney’s 2012 standing, Gallup’s resulted in a larger-than-average four-point shift. In particular, the finding mirrors problems in the 2008 New Hampshire primary, when Gallup’s likely voter model produced larger errors than un-adjusted data, according to a report by the American Association for Public Opinion Research.
Gallup said the 2012 election data were not sufficient to diagnose what was wrong withits likely voter model, but plans to test the accuracy of the model in gubernatorial elections this year in New Jersey and Virginia by comparing survey results with records of whether respondents actually voted.
2. Too many whites
Gallup’s pre-election surveys contained too many whites among the base sample of American adults and too few Hispanics and African Americans, a bias that had the potential for great impact given deep racial voting divisions. Newport said the way Gallup asked about race – a series of “yes/no” questions – resulted in a higher share of respondents identifying as multiple races than in government estimates and made weighting to correct distributions more difficult.
This year, Gallup implemented new questions about race that are similar to other public polls, a change that has “resulted in an impact on presidential job approval ratings,” according to Newport.
3. Sub-regional swings
Gallup’s final pre-election polls overrepresented voters in regions where Romney performed better. While Gallup sampled and weighted surveys to match each of the four regions’ population levels, Newport said certain sub-regions – in particular time-zones that interviewers called latest – were underrepresented throughout their election polling. “Better controls of calls within time zone sub-regions would have increased Obama’s margin over Romney by at least one percentage point,” Newport said. Gallup hasn’t changed their method, but says they are examining ways to better organize call strategies and produce a more accurate sample.
4. Secret landline phones
Gallup, like other pollsters, calls both landline and cellular phones in surveys to ensure nearly all Americans have a chance at participating. But in 2011, Gallup decided to begin only calling landline phone numbers listed in public directories, a practice that costs less, believing that people with unlisted landline phones could still be reached via cellular phones. But an experiment conducted this spring found that this method produced a sample that was older, less Democratic and less likely to approve of President Obama. As a result, Gallup plans return to calling both listed and unlisted landline numbers.
Gallup examined, but ruled out, a number of possible explanations for their 2012 poll miscues, described in their full report.
Scott Clement is a pollster with Capital Insight, the independent polling group of Washington Post Media. Pollster Peyton M. Craighill contributed to this report.