By David Brown
Washington Post Staff Writer
Monday, January 12, 2009
In our information-crazy, never-out-of-touch world, it's becoming harder and harder to find out who we are and what we do.
That's the ironic truth facing epidemiologists around the country.
The popularity of cellular telephones, an increasingly mobile population, rising expenses, flat budgets and new insights into ways people can answer a question differently depending on how it's asked -- all are conspiring to make health surveys more difficult.
In public health, pretty much everything depends on good data. Researchers and policymakers can't identify a problem, figure out whether it's serious and devise a strategy to fight it without first being able to count it. "If you can't measure something, you won't be able to change it" is an oft-heard aphorism.
How big a problem is obesity? Are restrictions on smoking changing people's habits? Is autism more prevalent than it was a decade ago? Is the recession affecting people's access to health insurance?
All are questions of national importance -- and none can be answered without unbiased surveys of a representative sample of the population.
Cellular telephones are perhaps the biggest threat to survey data that epidemiologists have confronted in years.
The National Center for Health Statistics reported that in the first half of last year, 16 percent of American adults lived in households that have only cellphones. This was up from 7 percent three years earlier, and rising rapidly.
The federal government's main tool for measuring the health habits of Americans, the Behavioral Risk Factor Surveillance System (BRFSS), uses the telephone to interview a nationwide sample of adults (470,000 this year). Historically, interviewers called only conventional telephones, as all but the 2 to 3 percent of households with no phones at all could be reached through them. But that's not remotely true anymore.
Surveyors, however, cannot just extrapolate from the land-line respondents. That's because studies have shown that people who have only cellphones are different from people who don't have them or use them only occasionally.
Young people, men and Hispanics are all more likely than the "average" American to have cellphones only. But those demographic factors don't explain everything. Even after they are taken into account by statistical means, cellphone-only users are different.
The BRFSS surveyors this year will include cellphone numbers in every state, with a goal of having 10 percent of the interviews done that way. But it's easier said than done.
Federal law requires that calls to cellphones be hand-dialed; it is illegal to use automatic dialers, which are standard tools for survey and polling firms. Furthermore, a huge fraction of "owners" of cellphone numbers are children ineligible for the health surveys. Once reached, some cellphone users are reluctant to talk at length because they have to pay for incoming calls.
Consequently, it takes roughly nine calls to working cellphone numbers to get one completed survey, compared with five calls to working land-line numbers, said Scott Keeter, a polling expert at the Pew Research Center for the People and the Press, an independent opinion research group. Further, an interview conducted with someone who uses a cellphone costs 2 1/2 times as much as an interview with someone on a conventional phone. In addition to higher labor costs, most surveys now reimburse cellphone users for their minutes, either in cash or through credits to online merchants such as Amazon.com.
People's willingness to answer questions has also been affected by the barrage of phone calls, many unsolicited, they get every day. The response rate in public opinion polls has fallen from about 60 percent two decades ago to 25 percent now, according to Keeter.
Government-sponsored health surveys have fared better. The "cooperation rate" for the BRFSS in 2007 was 72 percent, the same as in 1994. The smaller, in-person National Health Interview Survey had a response rate of 87 percent in 2006. A decade earlier, it was 92 percent.
But the problem goes beyond changing technology and responsiveness. It turns out that people answer the same question differently depending on how you reach them -- a "mode effect."
For example, when a group of people with the same age, race and education are called on a conventional phone, 25 percent say they smoke, but on a cellphone 31 percent say they do. On a land line, 38 percent say they have been tested for HIV, while on a cellphone 54 percent say they have.
The reason for this is unclear. Ali H. Mokdad, an epidemiologist who until recently was at the federal Centers for Disease Control and Prevention and ran the BRFSS, speculates it may have something to do with the fact that people on land lines are usually at home, where they have a role and image to maintain even if they are answering in privacy.
"They are less likely to say something bad about their own behavior. It's like 'This is my house,' " he said.
A study published three years ago showed that when women were interviewed by phone, rather than in person, they tended to underreport their weight, and both men and women (but men more) tended to overreport their height. Both estimates were exaggerated -- women's weight lower and men's height higher -- compared with surveys in which height and weight were measured.
The study showed the importance of using measurements to determine the national prevalence of obesity, as interview data alone substantially underestimate it.
The internal architecture of a survey can also affect the results.
Where a question falls in a series of them can increase the likelihood of a certain answer, a phenomenon known as "differential item functioning." The ethnicity of the respondent may also make a difference. According to a study published in 2006, Hispanics are more likely to give answers at the extremes of numerical scales "because of a cultural value that associates extreme responses with sincerity."
In all, what seems like a fairly simple and straightforward task is in truth hard and messy.
"It's a bit like making sausage," said Christopher J.L. Murray, a physician and epidemiologist who heads the Institute for Health Metrics and Evaluation at the University of Washington.
"As soon as you start to explore how surveys are made," he said, "you begin to see how difficult it is to get consistent information at the population level over time."