Deja vu for New Hampshire polls?
By Jon Cohen,
In short, the pre-election polls before the New Hampshire primary in 2008 were a disaster. The numbers had anticipated a clear result: It would be a second, major win for then-Sen. Barack Obama over Sen. Hillary Rodham Clinton, who had been the overwhelming favorite for the Democratic nomination.
Instead, Clinton won by about two percentage points, setting off the polling industry’s biggest reevaluation since the infamous “Dewey defeats Truman” headlines of 1948.
With voters in the Granite State again set to cast ballots in the country’s first primary, are we in for a surprise similar to the one greeting poll watchers four years ago? After all, even before the 2008 debacle, The Washington Post dubbed the state “a snowy graveyard for pols and pollsters.”
A big shock is unlikely, at least at the top of the heap — where former Massachusetts governor Mitt Romney has a large, consistent lead — but that’s not necessarily because the polls are better this time around.
The New Hampshire polls in 2008 sent shock waves not mainly for their numerical inaccuracy — pre-primary polls are notoriously imprecise — but because most presaged the wrong winner. On average, the polls before the Iowa caucuses were just as off-target, but the final result lined up with expectations gleaned from most polling there.
And this year’s Iowa polls were no better than they were four years ago, with an average miss on the top two candidates equal to the Clinton and Obama differences in 2008.
After so many surveys got the winner wrong in New Hampshire last time, there was intensely negative attention on political polls. The American Association for Public Opinion Research — an industry organization — responded by creating a task force to dissect the data from New Hampshire and four other states. (Disclosure: I served on AAPOR’s executive council from 2009 to 2011.)
The resulting report — published in early 2009 — revealed no clear reasons for the New Hampshire slip-up. But a number of likely culprits emerged, including the compressed period between the Iowa caucuses and the New Hampshire primary, and a large influx of first-time voters that might have spoiled likely voter models tuned to past behavior or elections.
David Paleologos, who heads up the survey center at Suffolk University and is again conducting daily tracking polls in New Hampshire, says he is now using stricter criteria to identify probable voters. He used looser ones four years ago because of the great “fluidity between Democratic and Republican primary voters.” (New Hampshire voters who are not registered with a political party can opt to participate in either one.)
Without a meaningful Democratic primary this time around, one big polling hurdle is gone. Paleologos also now includes cellphones in Suffolk’s polls, as do some others. The AAPOR report said the lack of cellphone samples was unlikely to have caused the errors in 2008, but the proportion of New Hampshire adults who have only a cellphone has about doubled since then.
But the biggest difference this year is contextual — Romney’s advantage over his rivals is about double Obama’s best from four years ago. The errors would have to be that much greater to spoil most prognostications about the No. 1 spot.
Although polling in general might survive public shame in New Hampshire this year, what hasn’t changed in political polling is a continued lack of basic disclosure, pointing to continued risk.
AAPOR’s main finding about the 2008 polls focused on transparency, not polling acumen. Delayed and incomplete information about 2008 limited the researchers’ ability to draw conclusions about what happened, and why. And without such knowledge, the possibilities for similar errors remain.