The new numbers appear to fit a common storyline: Disappointed by President Obama’s unfulfilled promise of bipartisanship, Americans have retreated into their partisan corners and are now more bitterly divided than ever before.
But it’s the wrong story. And the reason involves an ongoing controversy in the polling community: whether online surveys are effective replacements for in-person and telephone surveys.
In 2012, the ANES collected data in two ways: its traditional in-person (face-to-face) interviews and a newly introduced, probability-based Internet sample conducted by GfK (formerly Knowledge Networks). Given that both the in-person and Internet surveys were designed to produce representative samples of Americans, we would expect the level of polarization to be similar in both surveys. It wasn’t.
Consider the figure below. It plots the absolute difference in people’s ratings of the two parties on a 0-100 scale. The larger this difference, the more differently the average person rates the two parties.
This measure of polarization was 6 points higher in the Internet sample than in the sample interviewed face-to-face. Compared to 2008, the Web sample suggests an 8-point increase in polarization. The face-to-face sample shows only a 2-point increase.
For another measure of partisan polarization, the share of the public whose ideology matches their partisanship, the appearance of rising polarization is entirely due to the Web sample. As the figure below shows, the percentage of ideologically consistent partisans in the face-to-face sample actually dropped by more than a point from 2008 to 2012. But in the Internet sample, it was nearly 5 points higher than in 2008.
Why would the two surveys produce such different findings? One possibility is that answering survey questions alone on a computer screen at your preferred pace may lead to different answers than responding verbally to questions posed by an interviewer in your home. A second possibility is that because the two surveys’ samples were drawn in different ways and had vastly different response rates (38 percent for the face-to-face survey and 2 percent for Internet survey, according to preliminary calculations by the ANES), the two samples differed in their political attitudes.
Differences between Internet samples and other survey modes are nothing new, of course. Internet surveys are a popular alternative to traditional face-to-face interviews and telephone surveys, due in large part to their lower cost and greater convenience. Considerable doubts remain about the validity of non-probability samples, including “opt-in” samples. But the ANES Web survey is not an “opt-in” sample. It was conducted by GfK using its regular pool of panelists recruited via traditional probability sampling—in this case a combination of random digit dialing and address-based sampling. In this case, there are clear substantive differences between two probability samples. And it is not clear which sample is right.
Figuring this out is important not only for survey geeks, but also for answering big questions about American public opinion. To many, President Obama’s first term and the 2012 campaign seemed only to polarize Americans further. But once you take the design of the survey into account, the new ANES data do not support that conclusion.