| || || ||
| ||ONLINE EXTRAS/What Americans Think|
Lights Out for Traditional Polls?
By Richard Morin
Washington Post Polling Director
Monday, May 15, 2000
Cyber-maven Andy Grove, the founder of Intel, famously predicted last year that in five years, all companies will be Internet companies or they won't be companies at all. Which makes me wonder: In five years, will all polls be Internet polls or will there be polls at all?
I fear that it's more than a rhetorical question. Response rates to traditional telephone surveys are disturbingly low. The plethora of telephonic intrusions into our homes is increasing, suggesting that more Americans will continue to say no to survey interviewers. And that means it may be getting close to bedtime for traditional polls.
But polling over the Internet, which has grown dramatically, poses problems of its own. Internet polls often rely on volunteer samples as opposed to true random samples of the population. And not everybody, particularly older people, minorities and the poor, is computer literate. Separately and together, these are serious reasons to question the validity (thus the usefulness) of most Internet polls.
Thus this bleak prospect: Traditional polls die a slow, painful death, and nothing replaces them. Oh, woe is us (and particularly me, since I pay the bills by doing polls for The Washington Post). Happily for us (and particularly me), these fears may be overblown. New polling techniques, new technologies andperhapsa new attitude toward practices currently scorned by many pollsters may mean there's a future for public opinion research.
Companies such as InterSurvey in California are developing hybrid techniques that combine the best of the old and the new. (The Washington Post is an investor in InterSurvey.) These firms draw random samples of the national population, contact them by phone, and then hook them up to the Internet. The result, at least in theory, is a pool of randomly selected Americans who can be sent questionnaires over the Internet and complete them in the privacy of their homes. Cooperation rates soar, sample skews are avoided, and a defensible random sample is produced.
But some polling experts advocate another way, one beyond traditional telephone surveys or techniques that use the Internet, to deliver surveys to true random samples of the population, an exceedingly expensive proposition. These rebels question the survey community's traditional reliance on pure probability samples. Call them the New School survey researchers.
One of the leading New Schoolers is among the reigning giants of the Old School: Humphrey Taylor of Harris Interactive, formerly Louis Harris and Associates. Writing in the International Journal of Market Research, he makes a reasoned plea for a reconsideration of survey researchersŐ worship of strict probability sampling techniques.
He begins by acknowledging the advantages of a true random sample. There is a sound statistical/mathematical theory to explain how and why it works, he writes. (It also means, in theory, that results are generalizable to the larger population, an assurance that no other form of respondent selection procedure allows.)
Well, theory's great. But Taylor also notes that with response rates (the percentage of people who complete an interview) below 50 percent, there's reason to doubt that most 'random' samples are truly random.
Instead, he argues, researchers should reconsider quota sampling techniques and volunteer samples, methods currently scorned by most pollsters. In a quota sample, researchers collect a sample by filling quotas that produce a representative sample of a given population. Thus if men comprise 48 percent of the country and women 52 percent, you interview slightly more women than men, though they don't have to be selected strictly at random. By setting representative quotas for years of schooling completed, race, income, region of the country and urban/suburban/rural residence, you should get a reasonable and representative sample of the country, albeit one that is not strictly random.
Quota sampling has been shown to work, and sometimes to work better (for example, in predicting election results) than probability sampling, Taylor writes. He acknowledges that "there is no good theory to explain how and why it works; therefore to statistical purists (particularly American ones), it is unacceptable. (That narrowly pedantic view, he says, is equally unacceptable.)
A second alternative, volunteer sampling, is even more frightening to purists but increasingly attractive to Internet pollsters. In one common Net version, a sample is recruited simply by posting a question that asks visitors to a Web page if they'd like to participate in a poll. If they click 'yes', a questionnaire appears or they are linked to a polling Web page. Thus they "volunteer" to participate. Random, it's not. Skewed? Usually, yes: If the recruiting question appears on, say, a sports bulletin board, disproportionately more guys will see it and respond; if it's posted on the Redbook magazine Web site, it will be seen by disproportionately more younger women.
Taylor argues that the key to making volunteer sampling respectable is huge samples recruited from various sources, which are entirely achievable via Internet recruiting. The theory goes something like this: If you post the question in enough places, you'll get enough old people, young people, black people, rich people, less well-educated people and poor people to use quota sampling to produce a representative sample of the population.
It doesn't matter if 1 percent of the 10,000 people who volunteered are black, just make sure that 11 percent of the national sample you select are, because about 11 percent of the adult population is black. You can either ask all the volunteers the poll questions and do a quota sample later to produce a representative sample, or select a representative sample and e-mail them questionnaires.
Harris Interactive is using a more nuanced version of these techniques to do online polling. In 1998, its pollsters attempted to estimate outcomes in races for governor and the U.S. Senate in 14 states. They correctly projected the winner in 21 of the 22 races, doing slightly better than traditional telephone polls, Taylor says.
Promising, indeed. But not conclusive. Much more work needs to be done before these techniques meet with general acceptance. But one thing is clear: We all must change. Or we will all be lost. And that may mean rethinking our disdain for quota and volunteer sampling techniques.
Archive: What Americans Think