December 19, 2013

There was a discussion today over Twitter, coming out of some blog posts, about whether polling or interviews is better for learning about voters. It’s a really good question. There’s an answer.

For reporters, or for anyone trying to understand politics, information is good. Quantitative (polling) or qualitative (interviews). More is better. But realize, accept and acknowledge the limitations of whatever information you’re working with.

Good reporters can do a lot with in-depth interviews. But they need to remember that even in a local election in a relatively small town, drawing too many generalizations from what a handful of people say can be extremely dangerous. And once it gets to extrapolating to hundreds of thousands or millions, the problem just gets worse. That doesn’t mean interviews are worthless! Far from it; there’s plenty that a great reporter can learn from ordinary-voter interviews that can be very, very, helpful to understanding aspects of an election, or of what people think about a policy. It’s just that without other information, it’s extremely easy to make dubious generalizations.

On the other hand, reporters can also do a lot with polling. But just because those columns of numbers seem so “real” doesn’t mean that they can be taken at face value. Interpreting polling takes at least as much skill as interpreting interviews. It’s not just the math of it — no, including a “margin of error” doesn’t come close to fulfilling a reporter’s obligations to the reader. Questions may elicit opinions which didn’t exist before someone answered the phone; in other cases, experienced poll-readers will realize that a polling snapshot isn’t likely at all to predict future opinions or, in the case of voting, future actions. And single polls, even when done well, are never as useful as good averages of multiple polls. There’s an enormous difference, in other words, between the averages of a dozen polls about the candidates the weekend before an election and a single poll asking for opinions about a complex policy that hasn’t been in news very much.

The most important thing for reporters is to learn about the limits of whatever information sources they use, and give a clear picture of exactly what they can say with confidence, what is merely suggestive and what should be pretty much ignored. And because each type of source has different strengths (and weaknesses), it’s almost always going to be better to use an all-of-the-above approach, if at all possible.