A new state election polling collaboration between the New York Times, CBS News and Internet pollster YouGov has drawn an unusual public rebuke from the leading organization of survey researchers, adding fuel to a fiery debate over what makes a poll "good" or "bad."
The American Association for Public Opinion Research (AAPOR) criticized the Times and CBS for its state polling with YouGov, saying the survey methods used by the polls has "little grounding in theory" and a lagging disclosure of methodological details required to assess the poll's quality. In addition, AAPOR chided the Times for removing its published set of poll reporting standards which had mostly barred the use of opt-in internet surveys — like those used by YouGov — by the newsroom and replacing it with a note explaining that it has begun a process to review its polling standards.*
"Unfortunately, due perhaps in part to the novelty of the approach used, many of the details required to honestly assess the methodology remain undisclosed," read the AAPOR statement in part. "This may be an isolated incident with the Times / CBS News providing more information on this effort in the coming weeks. If not, it is a disappointing precedent being set by two of our leading media institutions."
In response, CBS and the Times claimed to provide "great detail" about the new surveys in a brief statement, and defended the underlying methodology as "already well-known to the survey research community." (Make sure to read the Fix interview with Upshot editor David Leonhardt in which he explains the Times' decision in detail.)
The heart of the controversy centers on the usage of "opt-in" Internet panels rather than traditional "probability" methods which have underpinned public polling for decades. Probability-based polls aim to ensure that every person in the population has a chance of being selected to participate in the survey, a strategy that buttresses claims that the sample's results can be projected as estimates of the broader population (within a margin of sampling error). Non-probability surveys such as the CBS/Times/YouGov Battleground Tracker start with a pool of non-randomly recruited Internet survey-takers and use targeted selection and weighting, along with a statistical model of voting, to produce estimates of vote support in election races. The Times and CBS note they will continue to conduct probability-based telephone surveys in addition to Internet polls.
In a telephone interview, AAPOR President Michael Link said it was important for the organization to step into this situation because of the high visibility and credibility of polls offered by the New York Times and CBS News. He indicated that this is a bold step with a new polling methodology and there needs to be a solid understanding of how to evaluate these methods. At the same time, Link stressed AAPOR is not against new methods but stipulated that the "use of any new methods be conducted within a strong framework of transparency, full disclosure and explicit standards."
The decision to sponsor non-probability Internet polling marks a stark shift in standards for the New York Times. Prior to the collaboration, Times public standards warned of numerous problems with such surveys, including that opt-in surveys often pay respondents to complete polls, that Internet access is not "sufficiently widespread" and access is not evenly distributed across the public. The previous standards stated that for a poll to be "worthy of publication" it "must be representative, that is, based on a random sample of respondents." But the Times' standards changed after their new polling collaboration was published. The revised document indicates that decisions about what polls to publish will be made on a case-by-case basis while new standards are being produced.
AAPOR criticized this decision to adopt a new, piecemeal approach, without an explicit standard justifying their use. "Standards need to be in place at all times precisely to avoid the “we know it when we see it (or worse yet, ‘prefer it’)” approach, which often gives expediency and flash far greater weight than confidence and veracity."
*Full disclosure: Washington Post pollsters Peyton Craighill and Scott Clement are members of AAPOR. Neither was involved in AAPOR's statement regarding CBS News, New York Times and YouGov polls.