Our Polls Are on the Mark. I Think.

By Jon Cohen
Sunday, November 2, 2008

January, you may recall, was a rough month for the pollsters. All the polls showed Sen. Barack Obama poised to follow up his big win in the Iowa caucuses with a knockout blow to Sen. Hillary Rodham Clinton in the New Hampshire primary. But he lost, sending the 13 firms that did public pre-election polls there scrambling for explanations.

Could polling be similarly embarrassed this month, misjudging the last chapter of this epic presidential election? Thoughts of the Granite State jolt me and my fellow pollsters awake in the dead of night during these final days.

Sen. John McCain certainly says that the polls are misleading, arguing that most surveys have "consistently shown me much further behind than we actually are," as he put it last Sunday on NBC's "Meet the Press." Of course, blaming the polls is standard operating practice for trailing candidates, their partisans and contrarians everywhere. And of course, we won't make the same mistakes that led George Gallup to declare that Thomas Dewey had Harry Truman beat in 1948: We won't use outdated sampling techniques, and we won't assume that the race is over and stop polling. Nor is our polling window as absurdly small as the four nights between Iowa and New Hampshire. Even so, could McCain be onto something? In the latest Washington Post-ABC News tracking poll of likely voters, Obama has a nine-point lead, larger than Dewey's five-point margin in the late October Gallup poll in 1948. Other reputable national polls this year show similar or even larger Obama leads. But could we still make big mistakes? Can the polls be trusted?

As the polling director of The Washington Post, I get that question just about every day, even in less intense periods than this. Some question the scientific basis of polling, refusing to believe that interviews with hundreds or thousands of randomly selected respondents could accurately represent the opinions of many millions. Others see basic bias (refreshingly, these accusations come from all sides) or point out new wrinkles, such as the growing number of adults in the United States who have only cellular phones that pollsters mostly don't call. Added to the mix this year is a lingering skepticism about the accuracy of polling contests between white and black candidates -- doubts that persist despite decades of data suggesting that these polls perform no worse than others -- and heightened concerns about the way we define "likely voters."

I don't consider any of these fatal or even very serious problems, but that doesn't mean I'm immune to pollster's paranoia. We could all be wrong -- at least theoretically.

Simply put, we may be wrong about who is likely to vote on Tuesday. One of the trickiest parts of political polling is determining which of the people interviewed in pre-election surveys will really vote. It's relatively easy for us to identify such sharply delineated groups as the population of all adults living in the United States or even all registered voters, but the pool of actual voters is a group that exists at a single point in time, on Election Day (plus those casting ballots early and by mail).

Even a few days away from an election, that group remains an unknown population. Not everyone who says that they will vote will actually do so, in part because when asked about their intentions, people want to sound like good citizens. So pollsters develop models to whittle down their samples to account for people's tendency to overstate these things.

These "likely voter models" vary widely from pollster to pollster. This year, the Gallup Organization is publishing two models. Its "traditional" model factors in respondents' reports of whether they voted in previous elections to determine who is a "likely" voter. But Gallup's new, "expanded" model drops this requirement, putting more young and minority voters into the "likely voter" category.

In Washington Post-ABC polling, we ask a series of questions about whether and how people plan to vote, whether they have voted before and basic knowledge about the voting process. We then feed all this information into a range of models, corresponding to different levels of turnout. We report a single model, but only after assessing the quality and impact of all of them. Likely voter modeling is a craft, bolstered by science.

I'm also often asked about the rising use of cellphones. The number of people ditching their home telephones has spiked in recent years, with the highest percentages among young adults and nonwhites. Does this affect the polls? Probably not -- or at least not yet. The exclusion of cellphone users appears to have no more than a minimal effect on the results. But even if these voters turn out to have been systematically underrepresented in this year's polls, that would actually mean that Obama had an even larger lead, because these voters overwhelmingly back him over McCain. And both the Gallup and Washington Post-ABC tracking polls interview complementary samples of voters who have only cellphones to make sure that we're not missing something. (Few state polls include cellphone interviews.)

Others who doubt this year's polls raise the question of a "Bradley effect." This syndrome gets its name from the bid by Tom Bradley, then the African American mayor of Los Angeles, to become governor of California in 1982. He headed into Election Day with a big seven-point lead in the last publicly released poll, only to lose narrowly. Some attributed this startling result to a quiet form of racism that revealed itself only in the voting booth, and the 1982 case has been trotted out ever since to cast doubt on the accuracy of pre-election polls in contests between white and black candidates.

But there is good reason to doubt that racism was the cause of Bradley's defeat, and decades of polling in other campaigns with black candidates should mute some of the skepticism. No "Bradley effect" has shown up for years, and a new analysis by one Harvard University researcher, Daniel Hopkins, shows that any such effect that existed in African American politicians' contests in the late 1980s and early 1990s has now disappeared.

CONTINUED     1        >

© 2008 The Washington Post Company