If We Ask It Right, You Answer Well
Washington Post Staff Writer
Sunday, January 10, 1999; Page C1
If his current government job ends abruptly, President Clinton might think about becoming a pollster. Anyone who ponders the meaning of the word "is" has precisely the right turn of mind to track public opinion in these mindless, mindful times.
Never has polling been so risky – or so much in demand. Never have so many of the rules of polling been bent or broken so cleanly, or so often. Pollsters are sampling public reaction just hours – sometimes minutes – after events occur. Interviewing periods, which traditionally last several days to secure a solid sample, have sometimes shrunk to just a few hours on a single night. Pollsters have been asking questions that were taboo until this past year. Is oral sex really sex? (Yes, said 76 percent of those interviewed in a Newsweek poll conducted barely a week after the scandal broke back in January.)
"No living pollster has ever had to poll in a situation like this," said Michael Kagay, the editor of news surveys at the New York Times. "We're in uncharted territory." After all, Andrew Johnson had to deal with political enemies, but not pollsters. And Richard Nixon's resignation before impeachment meant that pollsters didn't have a chance to ask whether the Senate should give him the boot.
Clinton has it about right: Words do have different meanings for different people, and these differences matter. At the same time, some seemingly common words and phrases have no meaning at all to many Americans; even on the eve of the impeachment vote last month, nearly a third of the country, didn't know or didn't understand what "impeachment" meant.
Every pollster knows that questions with slightly different wording can produce different results. In the past year, survey researchers learned just how big and baffling those differences can be, particularly when words are used to capture public reaction to an arcane process that no living American – not even Strom Thurmond – has witnessed in its entirety.
Fear of getting it wrong – coupled with astonishment over the persistent support for Clinton revealed in poll after poll – spawned a flood of novel tests by pollsters to determine precisely the right words to use in our questions.
Last month, less than a week before Clinton was impeached by the House, The Washington Post and its polling partner ABC News asked half of a random sampling of Americans whether Clinton should resign if he were impeached or should "fight the charges in the Senate." The other half of the sample was asked a slightly different question: Should Clinton resign if impeached or should he "remain in office and face trial in the Senate?"
The questions are essentially the same. The results were not. Nearly six in 10 – 59 percent – said Clinton should quit rather than fight impeachment charges in the Senate. But well under half – 43 percent – said he should resign when the alternative was to "remain in office and stand trial in the Senate." What gives?
The difference appears to be the word "fight." America is a peaceable kingdom; we hate it when our parents squabble and are willing to accept just about any alternative – including Clinton's resignation – to spare the country a partisan fight. But when the alternative is less overtly combative – stand trial in the Senate – Americans are less likely to scurry to the resignation option.
Such a fuss over a few words. But it is just more proof that people do not share the same understanding of terms, and that a pollster who ignores this occupational hazard may wind up looking for a new job.
Think I'm exaggerating? Then let's do another test. A month ago, how would you have answered this question: "If the full House votes to send impeachment articles to the Senate for a trial, then do you think it would be better for the country if Bill Clinton resigned from office, or not?"
And how would you have answered this question: "If the full House votes to impeach Bill Clinton, then do you think it would be better for the country if Bill Clinton resigned from office, or not?"
The questions (asked in a New York Times/CBS News poll in mid-December) seem virtually identical. But the differences in results were stunning: Forty-three percent said the president should quit if the House sends "impeachment articles to the Senate" while 60 percent said he should quit if the House "votes to impeach."
What's going on here? Kagay says he doesn't know. Neither do I, but here's a guess: Perhaps "impeach" alone was taken as "found guilty" and the phrase "send impeachment articles to the Senate for a trial" suggests that the case isn't over. If only we could do another wording test. . .
Language problems have challenged pollsters from the very start of the Monica Lewinsky scandal. Among the first: How to describe Monica herself? The Washington Post's first survey questions referred to her as a "21-year-old intern at the White House," as did questions asked by other news organizations. But noting her age was potentially biasing. Highlighting her youthfulness conjured up visions of innocence and victimhood that appeared inconsistent with her apparently aggressive and explicitly amorous conduct with Clinton. In subsequent Post poll questions, she became a "former White House intern" of indeterminate age.
Then came the hard part: How to describe what she and Bill were accused of doing in a way that didn't offend, overly titillate or otherwise stampede people into one position or the other? In these early days, details about who did what to whom and where were sketchy but salacious. It clearly wasn't a classic adulterous love affair; love had apparently little to do with it, at least on Clinton's part. Nor was it a one-night stand. It seemed more like the overheated fantasy of a 16-year-old boy or the musings of the White House's favorite pornographer, Penthouse magazine publisher Larry Flynt. Piled on top of the sex were the more complex and less easily understood issues of perjury and obstruction of justice. After various iterations, we and other organizations settled on simply "the Lewinsky matter" – nice and neutral, leaving exactly what that meant to the imaginations (or memories) of survey respondents.
One thing is clear, at least in hindsight: Results of hypothetical questions – those that ask what if? – did not hold up in the past 12 months, said political scientist Michael Traugott of the University of Michigan. Last January, pollsters posed questions asking whether Clinton should resign or be impeached if he lied under oath about having an affair with Lewinsky. Clear majorities said he should quit or be impeached.
Fast forward to the eve of the impeachment vote. Nearly everybody believed Clinton had lied under oath about his relationship with Lewinsky, but now healthy majorities said he should not be impeached – a tribute, perhaps, to the White House strategy of drawing out (dare we say stonewalling?) the investigation to allow the public to get used to the idea that their president was a sleazy weasel.
Fortunately, pollsters had time to work out the kinks in question wording. Demand for polling produced a flood of questions of all shades and flavors, and good wording drove out the bad. At times, it seemed even to pollsters that there may be too many questions about the scandal, said Kathy Frankovic, director of surveys for CBS News. Through October, more than 1,000 survey questions specifically mentioned Lewinsky's name – double the number of questions that have ever been asked about the Watergate scandal, Frankovic said.
Polling's new popularity has attracted a tonier class of critic. In the past, mostly assistant professors and aggrieved political operatives or their bosses trashed the public polls. Today, one of the fiercest critics of polling is syndicated columnist Arianna Huffington, the onetime Cambridge University debating champ, A-list socialite and New Age acolyte. A few weeks ago, Huffington revealed in her column that lots of people refuse to talk to pollsters, a problem that's not new (except, apparently, to Huffington).
Actually, I think Huffington has it backward. The real problem is that people are too willing to answer poll questions – dutifully responding to poll takers even if they don't really have an opinion or understand the question that has been asked.
A famous polling experiment illustrates the prevalence of pseudo-opinions: More than 20 years ago, a group of researchers at the University of Cincinnati asked a random sample of local residents whether the 1975 Public Affairs Act should be repealed. About half expressed a view one way or another.
Of course there never was a Public Affairs Act of 1975. Researchers made it up to see how willing people were to express opinions on things they knew absolutely nothing about.
I duplicated that experiment a few years ago in a national survey, and obtained about the same result: Forty-three percent expressed an opinion, with 24 percent saying it should be repealed and 19 percent saying it should not.
But enough about the problems. In hindsight, most experts say that the polls have held up remarkably well. Within a month of the first disclosure, the public moved quickly to this consensus, as captured by the polls: Clinton's a good president but a man of ghastly character who can stay in the White House – but stay away from my house, don't touch my daughter and don't pet the dog.
"It is so striking. The public figured this one out early on and stuck with it," said Thomas E. Mann, director of governmental studies at the Brookings Institution. "If anything, the only changes were these upward blips in support for Clinton in the face of some dramatic development that was certain to presage his collapse."
Mann and others argue that public opinion polls may never have played a more important role in American political life. "This last year illustrates the wisdom of George Gallup's optimism about the use of polls in democracy: to discipline the elites, to constrain the activists, to allow ordinary citizens to register sentiments on a matter of the greatest public importance," Mann said.
Well, hooray for us pollsters! Actually, there is evidence suggesting that all the attention in the past year may have improved the public's opinions of opinion polls and pollsters. And why shouldn't they? These polls have had something for everyone: While Democrats revel in Clinton's high job-approval ratings and otherwise bulletproof presidency, Republicans can point to the equally lopsided majority who think Clinton should be censured and formally reprimanded for his behavior.
A few weeks ago, as bombs fell in Baghdad and talk of impeachment roiled Washington, pollster Nancy Belden took a break from business to attend the annual holiday pageant at her 10-year-old son's school. As she left the auditorium, the steadfast Republican mother of one of her son's classmates approached Belden and clapped her on the shoulder. "Thank heavens for you pollsters," she said.
"I was stunned. I was delighted," Belden laughed. "I've spent many years being beat up on by people who complain that public opinion polling is somewhat thwarting the political process, as opposed to helping it. Suddenly, people are coming up to me at parties and saying thanks for doing what you do. What a relief!"
Richard Morin is director of polling for The Washington Post. "What Americans Think" appears Mondays in The Washington Post National Weekly Edition. Morin can be reached at firstname.lastname@example.org.
© Copyright 1999 The Washington Post Company