By George Bishop
Sunday, February 3, 2008
C an we ever trust the polls again?
It was the question on all minds after the pollsters' stunning miscall of the New Hampshire Democratic primary last month. The oh-so-wrong predictions that Sen. Barack Obama would beat out Sen. Hillary Rodham Clinton in the Granite State suddenly made the whole polling enterprise seem a bit shaky. What's worse, the pollsters have had a bumpy ride all through this primary season, miscalculating margins and magnitudes of wins in Michigan, Nevada and South Carolina.
Does it matter? Well, yes, because we all rely on polls as one of our best sources of information about the actions, opinions and motives of the American electorate. But if the polls are getting things so wrong lately, then we have to wonder whether they're right about things other than the horse race -- about Americans' preferences and positions on policy issues that make a difference in how the country is run.
The truth is, a healthy skepticism about polls is always in order. The trouble with election polls is that they often seem to be telling us a lot more about a race than they really can. And almost always, they tell us much more about who will win than why.
In some sense, polls are the victims of their own success. Usually, they're pretty much on the money, making the outcome of presidential elections seem all too predictable. If you look closely at the historical record, you can't deny the outstanding accuracy of the final presidential polls. Yes, there was that infamous "Dewey Defeats Truman" disaster of 1948 -- the result of poor sampling methods and, more important, failing to poll right up to the election. But since then, the record has been remarkably good, apart from a few small inaccuracies, such as underestimates of Ronald Reagan's victory margin in 1980 and overestimates of Bill Clinton's in 1996.
So when polls fail to predict the winner or don't even come close, they violate our expectations, injecting an element of uncertainty -- and excitement -- into a race. After New Hampshire, the media seemed almost gleeful at the pollsters' comeuppance. "It's the Voters, Stupid," said Time magazine's Jan. 21 cover. Forget those so-called scientific surveys telling us what we're going to do. The voters had trumped the all-knowing pollsters.
But in fact, the polls' record in non-presidential elections, especially primary elections, has traditionally been less than stellar. In the 2000 New Hampshire primary, for example, the polls significantly underestimated the size of Sen. John McCain's victory on the GOP side and overestimated the size of Vice President Al Gore's on the other. Or recall the gross underestimate of George Wallace's support in the 1972 Democratic primary in Indiana, or the failure to catch Eugene McCarthy's surge in the 1968 New Hampshire Democratic primary.
Exit polls have had their problems, too. In 2004, they were way off the mark in forecasting a John Kerry presidency. Then there was the fallout from exit-poll-driven forecasts in Florida in 2000, which led competitive television networks to prematurely call a victory for Gore.
The polls took it on the chin again this year in New Hampshire, Nevada and the South Carolina Democratic primary, where they wildly underestimated Obama's winning percentage by an average of roughly 17 percent -- nearly twice the size of the misestimate for Clinton in New Hampshire. But are these failures symptomatic of an emerging problem or just an anomaly for the textbooks?
The recent botched forecasts might make some a little less confident than normal that the pollsters will get it right on Super Tuesday. As John F. Kennedy once remarked, "Success has many fathers, but failure is an orphan." But in the New Hampshire Democratic primary, failure had plenty of possible fathers. Clinton's near-tears, the candidates' ballot position and undetected racism were all mentioned as probable reasons for the debacle. So, most prominently, were voters changing their minds at the last minute and "late deciders" whom the pollsters' standard questions and likely-voter screens didn't detect. But if that was the cause, doesn't it say something about the flaws in the pollsters' methods? Otherwise, that's an all-purpose, after-the-fact loophole for what went wrong: The voters must have changed their minds. What else could it be? That's like heads you win, tails I lose.
The rub is that none of these accounts provides a good explanation for what's happened elsewhere. What went wrong in New Hampshire should have helped tell us something about how the polls would perform in the Michigan GOP primary, the Nevada caucuses and South Carolina: about who was more likely to vote at the last minute or change their mind at the last moment, and why. They didn't.
The polls did a bit better in Michigan than in New Hampshire, but they notably underestimated Mitt Romney's support by a margin not that different from the underestimate of Clinton's support the week before. But because there was no failure to call the winner in Michigan, such inaccuracies will tend to be overlooked, if not forgotten. Ditto for Obama's blowout of Clinton in South Carolina. All the polls nailed the winner, though they missed the magnitude of his victory by the proverbial mile.
An ad hoc committee of the American Association for Public Opinion Research is getting ready to review the pre-election polling performance in New Hampshire and South Carolina, as well as the Florida primaries and all the Super Tuesday primaries this week. Its report is bound to tell us something about the factors that may have contributed to New Hampshire and to some of the inaccuracies elsewhere. But in the end, there may be no smoking gun and no one-size-fits all explanation for the polls' performance. In the meantime, we'll hear a lot about how hard it is to determine the turnout of that elusive creature in U.S. politics, "the likely voter" and its not too distant relatives, the "undecideds" and the "last-minute deciders."
But here's the crux: In dutifully answering pollsters' survey questions, do voters really know how likely they are to vote? Do they actually know when they finally decided whom to vote for? And how do they know whether they're likely to change their minds at the last moment and vote for someone else?
Modern cognitive neuroscience would tell us that these are very tricky matters, that we're largely strangers to ourselves, and that we don't have direct access to the thought processes that govern our actions in the voting booth, or anywhere else for that matter. But illusions of cause and effect in public opinion polls, like other illusions, have a way of persisting, especially when we try to find out what issues or factors made the difference in an election.
When we ask survey respondents which issue or which candidate quality most influenced their votes, we shouldn't take their self-reports at face value. When they say that the economy, the war in Iraq, the candidate's experience or advocacy of "change" was most significant, they're not reporting causal influences. They're offering us plausible reasons, justifications or after-the-fact rationalizations for inaccessible, neurologically and emotionally driven decisions.
In the 2004 presidential election, the polls told us that "moral values" was a decisive factor in George W. Bush's reelection, but my analysis and that of others indicated that it was largely an illusion generated by the way the exit poll question was asked.
Along with demographic analyses of the electorate, such self-reports may provide some useful clues about what factors might have mattered, or what media messages and partisan spin voters may have been exposed to during the campaign, but not much else. It will tell us very little, psychologically, about why a given contest turned out the way it did.
So why do we care whether polls make the right prediction about a primary race or a presidential election? Because an accurate prediction not only confirms the scientific status of polling, it also helps justify the entire enterprise. If pollsters get the elections right, then this implies that they must get a lot of other things they ask about right, too -- issues, policies, public preferences -- plus or minus some margin of error. Unlike elections, those kinds of public opinion surveys have no Wednesday-morning reality check to tell us whether they're systematically underestimating public support for immigration reform, for example, in the same way that they underestimated Clinton's margin in New Hampshire or Obama's in South Carolina. So much depends on how questions about public policy are framed.
So if we get the elections wrong, can we trust the results of other surveys? If the kinds of problems the pollsters are currently experiencing become more common, we'll have to take a serious look at that question. But no one's going to be advocating getting rid of polls. After all, what would we do without them to kick around, especially in a campaign as heated and erratic as this one? Let's face it: We can't always trust them, but we can't live without them.
George Bishop is a professor of political science at the University of Cincinnati and the author, most recently, of "The Illusion of Public Opinion: Fact and Artifact in American Public Opinion Polls."
View all comments that have been posted about this article.