| || || ||
| ||ONLINE EXTRAS/What Americans Think|
Online Polling Done Right
By Richard Morin
Washington Post Polling Director
Monday, February 28, 2000
As readers of this column know, I hate online polls. Actually, I love to hate them. Some of the columns that I've most enjoyed writing have been based on the wacky results of online balloting, such as the People magazine website poll that named Howard Stern's sidekick "Hank the Angry, Drunken Dwarf" as one of the 50 most beautiful people in the world.
As readers of The Washington Post and the National Weekly Edition know, The Post recently sponsored a poll conducted by InterSurvey, a Palo Alto, Calif., firm that conducts surveys over the Internet. I wrote the story, with Post colleague Howard Kurtz, summarizing the results. And as visitors to the bulletin board of the American Association for Public Opinion Research (AAPOR) know, Howie and Ibut mostly me, and appropriately sohave been taking a wee bit of a pasting in recent days over that story, which ran last week, and that survey.
One angry e-mailer noted: "[T]he article itself provides a sad example of the extent to which the press will misrepresent polls and sampling error to its own advantage . . . . Since Mr. Kurtz spends much of his time describing how political ads stretch the truth, it is all the more depressing to see this under his own byline. And Rich Morin, an AAPOR member, should certainly know better." Then again, apparently not. Other AAPOR e-mailers raised many substantive questions about the characteristics of the sample (or is it a panel?), how to calculate margin of sampling error, and questions about the design itself.
I'll leave it to others to debate these issues. I will note one thing: The people behind InterSurvey are neither reckless nor methodologically naive. The company is headed by Doug Rivers and Norman Nie, both professors of political science at Stanford University. Rivers is a renowned sampling statistician. Nie was one of the principals behind SPSS, the statistical software program. His prior affiliations include the University of Chicago and the National Opinion Research Center.
So why my change of heart about this Internet poll? It's not, as one particularly caustic critic hinted, because The Post is an investor in InterSurveya fact that I should have disclosed in the story.
It's because InterSurvey represents the first and best attempt to apply rigorous survey methods to research over the Internet. It is an effort that should be carefully watched and responsibly critiqued, but also encouraged.
The major flaws of most Internet surveys are obvious. Most have to do with the characteristics of the sample. Nobody maintains a list of every e-mail address, so nobody can randomly pick e-mail addresses.
But the biggest problem is that not everybody has a home computer; fewer are on the Internet; fewer still use e-mail. Thus Net Nation looks very different from the population as a whole. It's somewhat younger, whiter, more male, better educated and more affluent than the nation as a whole (though less so now than a few years go). Thus any sample of Net users is not representative of the general population. And that means you can't generalize results from even the best Internet polls to the country as a whole. What good is that?
InterSurvey addresses those problems directly. InterSurvey draws a random sample of U.S. households using random-digit-dial techniques used in telephone surveys. All selected households are provided with free WebTV hardware and free Internet access. More than 25,000 people are part of this panel; InterSurvey expects to have 100,000 by the end of the year.
Surveys are sent over the Internet to respondents, who read the questions on their television screens, and answer them using the WebTV keyboard. For particular studies, subsamples are drawn at random from the panel. As Rivers noted in a posting to AAPOR members, "these samples are true probability samples with a sampling frame that includes all households, including households without computers. We do not use volunteers."
Are there potential problems? Of course, some eventually may prove lethal to this fledgling company. People aren't dropped from the sample pool after each survey. Thus some respondents may be interviewed several times over the course of a year. That raises the possibility of panel effects, which are changes in attitudes or behavior that occur simply because an individual is participating in a panel study. Rivers says past research into panel effects has been inconclusive. But they monitor their panel members and constantly refresh the potential respondent poll so as to avoid respondent fatigue.
So far, cooperation rates have been astonishing. Most people initially contacted agree to accept the equipment, and in two Post-Internet surveys, the return rate has averaged about 80 percent. The overall cooperation ratewhich factors in those who initially refused to accept the equipment and join the panelis comfortably over 50 percent, far better than on standard telephone surveys. Will these cooperation rates always be so high? Perhaps not. If InterSurvey hammers its respondent base with repeated questionnaires, chances are good that both the quality of the interviews and the participation rate will fall. It remains to be seen if InterSurvey can keep refreshing its panel at an acceptable ratea problem that will become more acute as its business grows.
And grow it will, I strongly suspect. One reason I hope it succeeds is that it offers researchers the opportunity to do things they presently cannot do over the telephone. In The Post's ad test, we showed a random sample of campaign ads being run by George W. Bush and John McCain and surveyed reaction. You can't show TV ads over the telephone.
Remember, take all of this with a grain of salt: The bosses of my boss have invested in this company. Time may prove InterSurvey to be a bust, brought low by unforeseen methodological problems or by market forces. That said, I personally find InterSurvey to be an exciting and useful tool. As with any new methodology, I expect bumps along the way. But the more I see it, and the more I use it, the more I like it.
Archive: What Americans Think