Why We Keep Getting Snowed by the Polls

An ad hoc committee of the American Association for Public Opinion Research is getting ready to review the pre-election polling performance in New Hampshire and South Carolina, as well as the Florida primaries and all the Super Tuesday primaries this week. Its report is bound to tell us something about the factors that may have contributed to New Hampshire and to some of the inaccuracies elsewhere. But in the end, there may be no smoking gun and no one-size-fits all explanation for the polls' performance. In the meantime, we'll hear a lot about how hard it is to determine the turnout of that elusive creature in U.S. politics, "the likely voter" and its not too distant relatives, the "undecideds" and the "last-minute deciders."

But here's the crux: In dutifully answering pollsters' survey questions, do voters really know how likely they are to vote? Do they actually know when they finally decided whom to vote for? And how do they know whether they're likely to change their minds at the last moment and vote for someone else?

Modern cognitive neuroscience would tell us that these are very tricky matters, that we're largely strangers to ourselves, and that we don't have direct access to the thought processes that govern our actions in the voting booth, or anywhere else for that matter. But illusions of cause and effect in public opinion polls, like other illusions, have a way of persisting, especially when we try to find out what issues or factors made the difference in an election.

When we ask survey respondents which issue or which candidate quality most influenced their votes, we shouldn't take their self-reports at face value. When they say that the economy, the war in Iraq, the candidate's experience or advocacy of "change" was most significant, they're not reporting causal influences. They're offering us plausible reasons, justifications or after-the-fact rationalizations for inaccessible, neurologically and emotionally driven decisions.

In the 2004 presidential election, the polls told us that "moral values" was a decisive factor in George W. Bush's reelection, but my analysis and that of others indicated that it was largely an illusion generated by the way the exit poll question was asked.

Along with demographic analyses of the electorate, such self-reports may provide some useful clues about what factors might have mattered, or what media messages and partisan spin voters may have been exposed to during the campaign, but not much else. It will tell us very little, psychologically, about why a given contest turned out the way it did.

So why do we care whether polls make the right prediction about a primary race or a presidential election? Because an accurate prediction not only confirms the scientific status of polling, it also helps justify the entire enterprise. If pollsters get the elections right, then this implies that they must get a lot of other things they ask about right, too -- issues, policies, public preferences -- plus or minus some margin of error. Unlike elections, those kinds of public opinion surveys have no Wednesday-morning reality check to tell us whether they're systematically underestimating public support for immigration reform, for example, in the same way that they underestimated Clinton's margin in New Hampshire or Obama's in South Carolina. So much depends on how questions about public policy are framed.

So if we get the elections wrong, can we trust the results of other surveys? If the kinds of problems the pollsters are currently experiencing become more common, we'll have to take a serious look at that question. But no one's going to be advocating getting rid of polls. After all, what would we do without them to kick around, especially in a campaign as heated and erratic as this one? Let's face it: We can't always trust them, but we can't live without them.

George.Bishop@uc.edu

George Bishop is a professor of political science at the University of Cincinnati and the author, most recently, of "The Illusion of Public Opinion: Fact and Artifact in American Public Opinion Polls."