Why We Keep Getting Snowed by the Polls

By George Bishop
Sunday, February 3, 2008

C an we ever trust the polls again?

It was the question on all minds after the pollsters' stunning miscall of the New Hampshire Democratic primary last month. The oh-so-wrong predictions that Sen. Barack Obama would beat out Sen. Hillary Rodham Clinton in the Granite State suddenly made the whole polling enterprise seem a bit shaky. What's worse, the pollsters have had a bumpy ride all through this primary season, miscalculating margins and magnitudes of wins in Michigan, Nevada and South Carolina.

Does it matter? Well, yes, because we all rely on polls as one of our best sources of information about the actions, opinions and motives of the American electorate. But if the polls are getting things so wrong lately, then we have to wonder whether they're right about things other than the horse race -- about Americans' preferences and positions on policy issues that make a difference in how the country is run.

The truth is, a healthy skepticism about polls is always in order. The trouble with election polls is that they often seem to be telling us a lot more about a race than they really can. And almost always, they tell us much more about who will win than why.

In some sense, polls are the victims of their own success. Usually, they're pretty much on the money, making the outcome of presidential elections seem all too predictable. If you look closely at the historical record, you can't deny the outstanding accuracy of the final presidential polls. Yes, there was that infamous "Dewey Defeats Truman" disaster of 1948 -- the result of poor sampling methods and, more important, failing to poll right up to the election. But since then, the record has been remarkably good, apart from a few small inaccuracies, such as underestimates of Ronald Reagan's victory margin in 1980 and overestimates of Bill Clinton's in 1996.

So when polls fail to predict the winner or don't even come close, they violate our expectations, injecting an element of uncertainty -- and excitement -- into a race. After New Hampshire, the media seemed almost gleeful at the pollsters' comeuppance. "It's the Voters, Stupid," said Time magazine's Jan. 21 cover. Forget those so-called scientific surveys telling us what we're going to do. The voters had trumped the all-knowing pollsters.

But in fact, the polls' record in non-presidential elections, especially primary elections, has traditionally been less than stellar. In the 2000 New Hampshire primary, for example, the polls significantly underestimated the size of Sen. John McCain's victory on the GOP side and overestimated the size of Vice President Al Gore's on the other. Or recall the gross underestimate of George Wallace's support in the 1972 Democratic primary in Indiana, or the failure to catch Eugene McCarthy's surge in the 1968 New Hampshire Democratic primary.

Exit polls have had their problems, too. In 2004, they were way off the mark in forecasting a John Kerry presidency. Then there was the fallout from exit-poll-driven forecasts in Florida in 2000, which led competitive television networks to prematurely call a victory for Gore.

The polls took it on the chin again this year in New Hampshire, Nevada and the South Carolina Democratic primary, where they wildly underestimated Obama's winning percentage by an average of roughly 17 percent -- nearly twice the size of the misestimate for Clinton in New Hampshire. But are these failures symptomatic of an emerging problem or just an anomaly for the textbooks?

The recent botched forecasts might make some a little less confident than normal that the pollsters will get it right on Super Tuesday. As John F. Kennedy once remarked, "Success has many fathers, but failure is an orphan." But in the New Hampshire Democratic primary, failure had plenty of possible fathers. Clinton's near-tears, the candidates' ballot position and undetected racism were all mentioned as probable reasons for the debacle. So, most prominently, were voters changing their minds at the last minute and "late deciders" whom the pollsters' standard questions and likely-voter screens didn't detect. But if that was the cause, doesn't it say something about the flaws in the pollsters' methods? Otherwise, that's an all-purpose, after-the-fact loophole for what went wrong: The voters must have changed their minds. What else could it be? That's like heads you win, tails I lose.

The rub is that none of these accounts provides a good explanation for what's happened elsewhere. What went wrong in New Hampshire should have helped tell us something about how the polls would perform in the Michigan GOP primary, the Nevada caucuses and South Carolina: about who was more likely to vote at the last minute or change their mind at the last moment, and why. They didn't.

The polls did a bit better in Michigan than in New Hampshire, but they notably underestimated Mitt Romney's support by a margin not that different from the underestimate of Clinton's support the week before. But because there was no failure to call the winner in Michigan, such inaccuracies will tend to be overlooked, if not forgotten. Ditto for Obama's blowout of Clinton in South Carolina. All the polls nailed the winner, though they missed the magnitude of his victory by the proverbial mile.

CONTINUED     1        >

© 2008 The Washington Post Company