By Richard Morin
By Richard Morin
Last year, political analyst Michael Barone outraged many pollsters when he suggested in The Weekly Standard that surveys may underrepresent conservative and Republican views because "conservatives are more likely than others to refuse to respond to polls, particularly those polls taken by media outlets that conservatives consider biased."
William Safire of The New York Times a few months later claimed that most media polls were "grievously misleading" in 1996, overstating Clinton's pre-election margin, which in turn reduced Republican turnout.
And academics are forever questioning whether the news media's desire to be first isn't forcing media pollsters to cut too many corners, cranking out sloppy surveys that mischaracterize the views of Americans.
These pundits and scholars are asking the same tough questions: Do the media polls misrepresent public opinion by interviewing samples that are too Democratic, too liberal, too white, too old or too something else?
For years, those questions were left unanswered.
Now there are answers. They come from a smart and ambitious experiment conducted last year by The Pew Research Center for The People & The Press to create the "perfect" poll and then compare its results to those obtained in a standard telephone survey.
Led by Andrew Kohut, center researchers asked the same questions in the same order in two different surveys. One survey used survey methodology used in many media polls, notably a five-day interviewing period that included making five attempts to make contact with potential respondents. The other poll identified as the "rigorous survey" but informally known in Pew offices as the "gold standard" featured unlimited attempts to reach respondents over an eight-week field period.
Additionally, letters asking for their participation were sent to each potential respondent before they were called. A $2 "incentive" was included in the letter. People who initially refused to participate in the poll were contacted twice more by telephone and asked to complete the survey. If they still refused, pollsters sent them a formal request for their participation by priority mail. The extra effort was designed to reach people who were frequently away from their homes, as well as to increase the percentage of potential respondents who ultimately completed the poll.
Pew researchers said comparing the two surveys should answer the question of whether Republicans or other groups are being missed in typical surveys. If they are, then the more rigorous poll should capture more of the underrepresented group and the results of the two polls should be significantly different.
The results are encouraging to those of us who do media polls for a living.
Pew researchers found that the demographics of the two samples varied little. The rigorous sample was slightly more affluent, a little better educated and contained slightly more whites than the standard survey. "But in most respects the two groups were the same and, more importantly, basically representative of the U.S. population as a whole," Kohut wrote.
Politically, few differences emerged. Those in the rigorous survey sample had slightly higher opinions of the Republican Party. "But on a number of questions including party identification and vote in the 1996 presidential election the rigorous sample was not more conservative than the standard sample," he wrote.
Some differences that did emerge seemed to be due to actual shifts in public opinion, and not methodological differences. Pew researchers found that 34 percent of those in the standard sample said the two political parties had been working together more to solve problems, and not "bickering and opposing one another." That's significantly less than the 40 percent of those in the rigorous sample who thought Republicans and Democrats were cooperating.
"But this difference may reflect an actual change in public attitudes over the course of the summer, following the passage of a balanced budget bill in July," Pew analysts noted. (In a separate national survey, Pew found in August that 43 percent of those interviewed said the two parties were getting along better.)
There were differences troubling differences. On two of the four race questions, white respondents in the rigorous survey expressed views that were far less sympathetic to blacks than respondents in the standard survey. Nearly two in three whites 64 percent in the rigorous survey said blacks who can't get ahead were responsible for their own condition, compared to 56 percent in the standard survey.
A similarly disturbing result emerged when researchers compared the attitudes of those who agreed to be interviewed and those who initially refused but were subsequently talked into participating in the poll.
More than one in five 22 percent of those who initially cooperated said they had a "very favorable" opinion of African Americans, compared to 15 percent of those who initially refused a pattern that researchers found was "similar for other minority groups as well."
"These differences offer a clue into what may be the biggest challenge facing pollsters who seek to accurately measure public opinion on racial issues," Pew researchers wrote. "People who are reluctant to participate in telephone surveys seem to be somewhat less sympathetic to blacks and other minorities than those willing to respond to poll questions. This suggests that to increase the accuracy of surveys that focus extensively on racial issues, pollsters need to make an extra effort to obtain interviews with people who initially refuse to cooperate."
The Pew experiment also challenged the theory that people who don't participate in polls refuse because they simply don't like or trust surveys.
Researchers found no differences in attitudes toward public opinion polls between the two samples. Nor were there differences in views among those who participated the first time they were asked and those who initially refused.
Two out of three in each sample said most polls work "for the best interest" of the public even though similarly large majorities "doubted that a random sample of 1,500 to 2,000 people could 'accurately reflect the views' of the nation." And three in four said they would participate in a survey again.
Richard Morin is director of polling for The Washington Post. "What Americans Think" appears Mondays in The Washington Post National Weekly Edition. Morin can be reached at firstname.lastname@example.org.
© Copyright 1998 The Washington Post Company