By Richard Morin
By Richard Morin
You get interesting answers when you ask a provocative question to people who ask questions for a living. At least that was the experience of the editors of Public Perspective when they recently asked nine top pollsters to write short essays in response to this question: What are the greatest challenges confronting public opinion research? Their answers will appear in the next issue of Public Perspective, published by the Roper Center for Public Opinion Research at the University of Connecticut.
"We expected answers at once candid and insightful," wrote Everett Carll Ladd, executive director of the Roper Center. "We were not disappointed."
Kathy Frankovic, director of polling at CBS News, says America's survey researchers squarely face their profession's biggest problem every time they look in a mirror. "Our greatest threat ... is the hardest to fight it's our own arrogance. Even when we know our methods cannot produce precision, we allow those who read or use our results to think they do.
She suggests that America's pollsters take off their wizard's hats and stop pretending our surveys are more accurate than they actually are or face further alienating a public already broadly suspicious of public opinion surveys.
"If we believe our own self-promotion today that as 'experts' we have developed precision instruments for measuring public opinion infallibly we risk undermining our real accomplishments and weakening our profession's well-deserved influence and respect," she wrote.
Frank Newport, editor in chief of the Gallup Poll, suggests the biggest challenge that researchers face is finding new and improved ways to disseminate survey results accurately and completely to the public. He notes that the relative ease in which surveys can be conducted and the intense interest of the media in surveys are both blessings and curses.
"Unlike other scientific endeavors, public opinion polls can be conducted with relatively little investment, at low cost, and quickly," Newport wrote. "Studies are conducted and released essentially without review or context. Media outlets often publish public survey results as received, without scrutiny, and assuming that designation of the source and sponsor are enough to absolve them (the media) from responsibility for the report's contents."
To solve the problem, he suggests more careful and rigorous review of poll results by journalists before the results are published. Some questions he thinks journalists should ask before reporting the results of a poll: "How [do] the findings compare to other poll results on the same topic, how was this poll done, and what criteria can be used to evaluate it?"
Howard Schuman, research scientist and professor emeritus at the University of Michigan, agrees that in polling, God is in the details. Polling's biggest challenge, he notes, is to acknowledge and attempt to address rigorously the myriad problems that can confound polls.
"Experienced survey professionals know that poll results are shaped by subtle aspects of questioning, by the nature of interviewing, by sample bias as well as sampling error, and by many other factors typically ignored by the public and, more importantly, by those wishing to use survey data to their advantage," Schuman wrote.
He also challenged survey researchers to ask tough questions of themselves and of their data: What have we measured and how can it best be interpreted "in the larger world of which surveys are one abstracted part."
Humphrey Taylor, chairman and chief executive officer of Louis Harris and Associates, sees new opportunities, new responsibilities and new problems in the spread of polling around the world.
In many emerging democracies, polling has become a powerful check against election fraud, and pollsters "to their surprise ... find themselves in an expected role as defenders of civil rights and bastions of democracy," noting that it's "harder to steal elections when honest and accurate pre-election opinion polls and exit polls show someone else well ahead."
At the same time, this role "puts honest pollsters at risk from those who want to corrupt the political process. Those who play along get rich; those who don't may suffer."
Last year, Taylor says, he traveled to Mexico and discussed polling in that country's presidential election with Mexican survey researchers. These weren't always pleasant conversations. He was told of "wildly inaccurate" pre-election polls in which the results were manipulated, and surveys that were never conducted. ("Some clients will pay handsomely for these phony polls," he notes.)
Taylor also says Mexican pollsters complained that polling firms that provided clients with numbers the clients didn't like sometimes didn't get paid or found their contracts for future surveys were abruptly terminated.
Other researchers say polling faces new challenges from new methodologies and strategies. Warren Mitofsky, president of Mitofsky International, says pollsters should ask "if there is a better way" to do the "ordinary things we do."
He points to the controversial methods used by John Zogby to track the 1996 presidential race, which included weighting results to party identification numbers and his use of daytime interviewing. Though roundly criticized by many pollsters, Zogby's final pre-election poll for Reuters was the most accurate of all the media tracking polls.
While I would not, at this time, advocate designing a political poll the way these surveys are designed ... it seems clear that they are doing something right, even though it differs from currently accepted practices," Mitofsky says. Zogby wrote that the biggest challenge facing pollsters is "harnessing the Internet for accurate, credible survey research." While still "too small and ungainly to produce any representative sampling," his research suggests that's changing and pollsters had better be ready.
Although there's a plethora of problems, er, challenges, facing pollsters at home and abroad, not one of these top pollsters was predicting the demise of polling, at least any time soon.
That's understandable, wrote Michigan's Schuman. "The power of the sample survey method is so great, so fully accepted, and so difficult to replace by any other method that even the most egregious blunders committed in its name have little or no effect on its further use."
Richard Morin is director of polling for The Washington Post. "What Americans Think" appears Mondays in The Washington Post National Weekly Edition. Morin can be reached at firstname.lastname@example.org.
© Copyright 1998 The Washington Post Company