Polls are coming faster and faster now — and so are complaints about them, whether it’s Republican complaints about the ratio of Democrats to Republicans in the new Post/ABC poll showing Obama ahead by three points nationally, or Democratic complaints about the likely voter screens in a Gallup/USA Today poll that found Mitt Romney up by five points in the “swing states.”
That last one may be new to you. Barack Obama pollster Joel Benenson today charged that Gallup’s likely voter screen was producing an electorate which, among other (alleged) improbabilities, had Obama and Mitt Romney tied among women in swing states. Benenson said that on average, “Obama leads among women by 10.3 points.”
So here’s a refresher about why the “unskewed” complaint is nonsense and what to make of the likely voter screen complaint.
First, the nonsense. Republicans this cycle have focused on party identification within polling samples, dismissing those which have the “wrong” mix. There’s just nothing to it. Party identification in the electorate is an unknown, just as vote choice is an unknown. Pollsters should not be weighting their polls for any particular balance of those who call themselves Democrats and Republicans because there’s no way of knowing what the electorate actually looks like at any given moment, or what it will look like on Election Day. We don’t know what voters call, or will call, themselves.
Why does the percentage of party identifiers vary in different polls? Other than random sampling variation, it’s probably because of two things. One is that there are a lot of people who behave like party loyalists but only sometimes think of themselves as “Democrats” or “Republicans.” In other words, their behavior doesn’t change, but whether they’ll show up in surveys as party identifiers does — and it probably changes depending on news about their party. The other is that as enthusiasm grows or wanes for a candidate, partisans move in or out of “likely voter” screens.
Basically, a poll that says a candidate will do better will, and should, also find that a larger percentage of the electorate belongs to that candidate’s party. The poll may be wrong, but a “skewed” party ratio isn’t evidence that it is.
What about the Obama campaign’s complaint about Gallup’s likely voter screen? It is possible that a likely voter screen can be biased or wrong. But there’s no way of knowing whether it is or isn’t until Election Day. And I certainly wouldn’t recommend ignoring Gallup polls. The problem here is that everybody knows that some registered voters won’t actually show up at the polls, but there’s just no reliable way to figure out who will and who won’t (no, asking people doesn’t work; they lie. Sorry). Each polling outfit has a method of trying to guess who will vote and who won’t; as the pollsters themselves will tell us, it’s far more of an art than a science — which means it could introduce systematic bias into their numbers. The problem is there’s no way of knowing which pollster’s likely screen is best.
One thing we do know will produce inaccurate polls: we should expect random sampling effects to show up in poll results, just as we should expect that if we flip a coin a hundred times we’ll sometimes get 55 heads and sometimes get 55 tails.
So knowing all of that, what should we do? Ignore as best as possible individual polls, and follow the sophisticated polling aggregates. I strongly recommend Mark Blumenthal’s HuffPollster and Nate Silver’s FiveThirtyEight. If one poll gets it wrong, the effect will be drowned out in the averages. Using only the poll-of-polls will keep everyone from the natural tendency to focus more on the polls we like and ignoring the others. If you want to know who is winning, that’s the best bet.