The Washington Post

How the polling establishment smacked down the New York Times

The sun peaks over the New York Times Building in New York August 14, 2013. REUTERS/Brendan McDermid

A new state election polling collaboration between the New York Times, CBS News and Internet pollster YouGov has drawn an unusual public rebuke from the leading organization of survey researchers, adding fuel to a fiery debate over what makes a poll "good" or "bad."

The American Association for Public Opinion Research (AAPOR) criticized the Times and CBS for its state polling with YouGov, saying the survey methods used by the polls has "little grounding in theory" and a lagging disclosure of methodological details required to assess the poll's quality. In addition, AAPOR chided the Times for removing its published set of poll reporting standards which had mostly barred the use of opt-in internet surveys — like those used by YouGov — by the newsroom and replacing it with a note explaining that it has begun a process to review its polling standards.*

"Unfortunately, due perhaps in part to the novelty of the approach used, many of the details required to honestly assess the methodology remain undisclosed," read the AAPOR statement in part. "This may be an isolated incident with the Times / CBS News providing more information on this effort in the coming weeks. If not, it is a disappointing precedent being set by two of our leading media institutions."

In response, CBS and the Times claimed to provide "great detail" about the new surveys in a brief statement, and defended the underlying methodology as "already well-known to the survey research community." (Make sure to read the Fix interview with Upshot editor David Leonhardt in which he explains the Times' decision in detail.)

The heart of the controversy centers on the usage of "opt-in" Internet panels rather than traditional "probability" methods which have underpinned public polling for decades. Probability-based polls aim to ensure that every person in the population has a chance of being selected to participate in the survey, a strategy that buttresses claims that the sample's results can be projected as estimates of the broader population (within a margin of sampling error). Non-probability surveys such as the CBS/Times/YouGov Battleground Tracker start with a pool of non-randomly recruited Internet survey-takers and use targeted selection and weighting, along with a statistical model of voting, to produce estimates of vote support in election races. The Times and CBS note they will continue to conduct probability-based telephone surveys in addition to Internet polls.

In a telephone interview, AAPOR President Michael Link said it was important for the organization to step into this situation because of the high visibility and credibility of polls offered by the New York Times and CBS News. He indicated that this is a bold step with a new polling methodology and there needs to be a solid understanding of how to evaluate these methods. At the same time, Link stressed AAPOR is not against new methods but stipulated that the "use of any new methods be conducted within a strong framework of transparency, full disclosure and explicit standards."

The decision to sponsor non-probability Internet polling marks a stark shift in standards for the New York Times. Prior to the collaboration, Times public standards warned of numerous problems with such surveys, including that opt-in surveys often pay respondents to complete polls, that Internet access is not "sufficiently widespread" and access is not evenly distributed across the public. The previous standards stated that for a poll to be "worthy of publication" it "must be representative, that is, based on a random sample of respondents." But the Times' standards changed after their new polling collaboration was published. The revised document indicates that decisions about what polls to publish will be made on a case-by-case basis while new standards are being produced.

AAPOR criticized this decision to adopt a new, piecemeal approach, without an explicit standard justifying their use. "Standards need to be in place at all times precisely to avoid the “we know it when we see it (or worse yet, ‘prefer it’)” approach, which often gives expediency and flash far greater weight than confidence and veracity."

*Full disclosure: Washington Post pollsters Peyton Craighill and Scott Clement are members of AAPOR. Neither was involved in AAPOR's statement regarding CBS News, New York Times and YouGov polls.

The Freddie Gray case

Please provide a valid email address.

You’re all set!

Campaign 2016 Email Updates

Please provide a valid email address.

You’re all set!

Get Zika news by email

Please provide a valid email address.

You’re all set!
Show Comments
Republicans debated Saturday night. The South Carolina GOP primary and the Nevada Democratic caucuses are next on Feb. 20. Get caught up on the race.
The Post's Dan Balz says...
Rarely has the division between Trump and party elites been more apparent. Trump trashed one of the most revered families in Republican politics and made a bet that standing his ground is better than backing down. Drawing boos from the audience, Trump did not flinch. But whether he will be punished or rewarded by voters was the unanswerable question.
GOP candidates react to Justice Scalia's death
I don't know how he knows what I said on Univision because he doesn't speak Spanish.
Sen. Marco Rubio, attacking Sen. Ted Cruz in Saturday night's very heated GOP debate in South Carolina. Soon after, Cruz went on a tirade in Spanish.
The Fix asks The State's political reporter where the most important region of the state is.
The State's Andy Shain says he could talk about Charleston, which represents a little bit of everything the state has to offer from evangelicals to libertarians, and where Ted Cruz is raising more money than anywhere else. In a twist, Marco Rubio is drawing strong financial support from more socially conservative Upstate. That said, Donald Trump is bursting all the conventional wisdom in the state. So maybe the better answer to this question is, "Wherever Trump is."
Past South Carolina GOP primary winners
South Carolina polling averages
Donald Trump leads in the first state in the South to vote, where he faces rivals Ted Cruz and Marco Rubio.
South Carolina polling averages
The S.C. Democratic primary is Feb. 27. Clinton has a significant lead in the state, whose primary falls one week after the party's Nevada caucuses.
67% 22%
The complicated upcoming voting schedule
Feb. 20

Democrats caucus in Nevada; Republicans hold a primary in South Carolina.

Feb. 23

Republicans caucus in Nevada.

Feb. 27

Democrats hold a primary in South Carolina.

Upcoming debates
Feb 25: GOP debate

on CNN, in Houston, Texas

March 3: GOP debate

on Fox News, in Detroit, Mich.

March 6: Democratic debate

on CNN, in Flint, Mich.

Campaign 2016
Where the race stands
Most Read



Success! Check your inbox for details.

See all newsletters

Close video player
Now Playing

To keep reading, please enter your email address.

You’ll also receive from The Washington Post:
  • A free 6-week digital subscription
  • Our daily newsletter in your inbox

Please enter a valid email address

I have read and agree to the Terms of Service and Privacy Policy.

Please indicate agreement.

Thank you.

Check your inbox. We’ve sent an email explaining how to set up an account and activate your free digital subscription.