By Behind The Numbers
Sunday, May 17, 2009
Wondering where all the polls have gone? For Yes, press 1.
After an election cycle with seemingly ubiquitous polls, readers of The Washington Post have recently seen none purporting to show the status of the hotly contested Democratic primary for Virginia governor.
The contest is only a month away, and The Post has been covering the campaign and is sponsoring a candidate debate Tuesday; so why no reporting on the polls?
In fact, there have been three recent, publicly released polls about the battle for the Democratic nomination. One had Terry McAuliffe, former chairman of the Democratic National Committee, with a double-digit lead, another had McAuliffe and former delegate Brian Moran running neck and neck. All had a common finding that there are scads of undecided voters. But The Post didn't publish reports on any of the polls.
Polls vary greatly in quality, and just because an organization produces a number does not mean that it is reliable, or that it's newsworthy.
Our responsibility is to scrutinize the data we report as carefully as we do the sources we quote in stories. By publishing numbers of uncertain quality or ones lacking essential context, we amplify those findings, and risk misleading you.
None of the recent polls in the Virginia governor's race meet our current criteria for reporting polls: Two primary ones were by Interactive Voice Response, commonly known as "robopolls," and the third was a partial release from one of the candidates eager to change the campaign story line.
The basic test of any poll is how reliably the opinions of those surveyed reflect those of everyone in the population of people that interests us, in this case prospective voters in the June 9 Democratic primary for governor.
Estimating how many and what types of people might vote a month out from a primary is often a monumental task, particularly so this year in Virginia. Will Democratic turnout be in line with the approximately 115,000 who voted in the party's lieutenant governor primary four years ago? The 150,000 or so voters who cast ballots in the 2006 Democratic primary for U.S. Senate? Or the nearly 1 million voting in last year's battle between Barack Obama and Hillary Rodham Clinton? Right now, turnout estimates range from very low to low.
Joe Lenski, executive vice president of Edison Research, the group that conducts the national exit poll for the consortium of television networks and the Associated Press, puts the problem bluntly: "Low turnout races with unpredictable voting patterns are notoriously difficult to accurately poll. The Virginia democratic primary would definitely qualify since there are so few past primaries to compare to and the turnout is likely to be very low."
We also avoided using the exact findings from the internal poll conducted by the pollster for one of the campaigns. Partisan pollsters tend to release results favorable to their candidate and typically lack basic elements of disclosure that we require.
Given the great complexity in determining "likely voters" in the upcoming electoral clash, extra care should taken to gauge whether people will show up to vote. Unfortunately, polls that use recorded voice prompts typically take less care than polls conducted by live interviewers.
Robopolls have limitations. People are generally less tolerant of long interviews with computerized voices. One recent Virginia robopoll asked six questions about the governor's race; the other asked four. The last time The Post polled a Virginia gubernatorial election, in October 2005, the questionnaire included dozens of queries about the race, as well as a series of "screening" items to determine who might actually turn out to vote.
Lost in the brevity is much, if any, substance. Neither of the two in Virginia asked about the top issues in the race, what candidate attributes matter most or anything about the economy. Without this essential context, these thin polls offer little more than an uncertain horse race number. In understanding public opinion, "why" voters feel certain ways is crucially important.
Like most media pollsters, campaign pollsters use other methods to add depth. Asked if he had used the technology, Joel Benenson, Obama's chief pollster, said, "No and I wouldn't. [Automated] polls are typically very short and campaign polls tend to be very strategic and longer. I think live contact helps keep voters engaged on these polls." Bill McInturff, lead pollster on John McCain's 2008 presidential campaign, said he does not use robopolls for campaigns.
The internal Moran poll that his campaign put out to counter the notion that McAuliffe had opened up a lead included 39 questions; four were released.
As a group, robopolls have a growing number of adherents who point to evidence that the method has shown "success" at coming close to election results. "I have an impression that [automated] preelection polling has been surprisingly accurate in recent years," said Jon Krosnick, a Stanford University professor.
There's a brewing battle among survey researchers about whether polls can be judged by how they perform in the final stages of campaigns. But until more research shows the validity of robopolls, we will continue to steer a careful path in our reporting.
-- Jon Cohen