The Washington Post
Navigation Bar
Navigation Bar

Related Items
  • Archive

  • Polls Section

  •  
    What America Thinks
    If You Can't Trust Pollsters...

    By Richard Morin
    Washington Post Polling Director
    Monday, July 12, 1999

    Uh, oh. We pollsters have a problem: Most people don't find us a particularly credible source of information when we speak on public issues, according to a team of researchers who are studying who Americans find credible – and why.

    Ronald Hinckley of Research/Strategy/ Management and political scientist Robert Y. Shapiro of Columbia University found that pollsters ranked 34th out of 44 sources of public-policy information in a national survey of 1,000 randomly selected adults.

    The researchers tested the credibility of 44 separate professional groups, including "national religious leader," "president of a large corporation" and "famous entertainer." For each, respondents were asked to rate "how believable, or unbelievable, do you think they are when they speak out on a public issue: very credible; somewhat credible; not so credible; or not at all credible."

    Answers were then converted into an index that ranged from 0 (not at all credible) to 100 (very credible), with 50 representing "indifference," researchers reported in a paper presented at the recent convention of the American Association for Public Opinion Research.

    So how'd we do? Not well. The average score of those groups included in the survey was 61.6; pollsters finished with a credibility score of 55.9. That's better than "candidate for public office" (53.1), union leaders (53), "famous athlete" (52.1) and TV or radio talk show host (46.6), among others.

    For the record, Supreme Court justices were the most credible of any group, with a score of 81.3; talk show hosts finished dead last.

    In This Corner

    The ongoing debate over Internet polling is heating up again. This time, two of the biggest names in survey research have squared off in the latest issue of Public Perspective to challenge and defend online polling.

    Humphrey Taylor, chairman of Louis Harris & Associates, exudes the exuberance of those who are doing pioneering work in online surveys in his piece "Heady Days Are Here Again" while Warren Mitofsky, president of Mitofsky International, takes a dimmer view of such efforts in the article "Pollsters.com."

    "We believe that online research will be a huge part of the survey research industry's future," write Taylor and his co-author, George Terhanian, director of Internet research for Harris Black International (HBI). "This is an unstoppable train, and it is accelerating. Those who don't get on board run the risk of being left far behind."

    They freely acknowledge that online samples are not random. (Technically, they're panels, not true samples.) Rather, respondents have been recruited from a variety of online sources. HBI asks people at various Web pages to send them their e-mail address if they want to participate in an online poll. Then they adjust their panel to compensate for the fact that the recruited sample may not accurately represent the electorate. (For the record, this weighting scheme is called "propensity score adjustment," which is too complex to explain here. They describe it as a "single, summary measure that represents the probability of belonging to one group rather than another.")

    Putting the methodology aside, the proof is in the polling. HBI's own test of online polling came last year in 22 Senate and gubernatorial races in 14 states; it got 21 of 22 winners right. More importantly, HBI came very close to matching the results of the final round of telephone polls conducted in each of those states.

    In terms of the overall spread – the difference between the winner's and loser's share of the vote – the average error was 6.8 percent in the final online poll, compared with 6.2 percent for the final telephone surveys.

    The average candidate error – the difference between the candidate's share of the vote in the poll and what they actually received on election day – in the final online survey was off by 4.4 percentage points in the Internet poll and 5 percent in the final telephone surveys. Mitofsky remains unimpressed.

    He does allow that the "performance of the Internet polls was fair." But he notes that one-third of the final online polls (seven of the 22) had an error that was larger than the margin of victory. And he notes that 29 of the 44 surveys were more accurate than the Internet polls conducted in those states, while 15 were less accurate and five had the same error.

    Beyond the tally of wins, losses and ties, Mitofsky notes that there remains no complete list of e-mail addresses "or a scheme for sampling them." Therefore all Internet poll samples must come from recruited or otherwise assembled panels, which means they will continue to have all the problems of any panel: too many of one type of person (white male computer geeks, for example) and too few of another type (technologically challenged seniors, for example).

    "What bothers me is the willingness to discard the use of sampling frames as a means of selecting a sample and then the feeble attempts at manipulating the resulting bias," Mitofsky writes. "That undermines the credibility of the survey process."

    Portent of Things to Come?

    Summer's here and those folks at Gallup couldn't resist having a little fun. In its latest national survey, Gallup asked a random sample of Americans who was better looking: Vice President Al Gore or Texas Gov. George W. Bush.

    The results: A dead heat. Gore was the choice of 38 percent of those interviewed, while 37 percent thought Bush was easier on the eyes, with the remainder perhaps too startled by the question to venture an answer.

    Richard Morin is director of polling for The Washington Post. "What Americans Think" appears Mondays in The Washington Post National Weekly Edition. Morin can be reached at morinr@clark.net .

    © Copyright 1999 The Washington Post Company

    Back to the top

    Navigation Bar
    Navigation Bar