"This is a serious issue," said Bruce D. Meyer, a professor at the Harris School of Public Policy at the University of Chicago. "The truth is that these government programs do a lot more to fight poverty than the survey suggests they do."
Meyer, whose research has long focused on the eroding quality of surveys in the United States, recently took a closer look at the numbers. Interested in seeing what would happen if more accurate data were used in place of the CPS, he conducted an experiment. He, along with Nikolas Mittag, a researcher at the Economics Institute of the Academy of Sciences of the Czech Republic, pitted two separate administrative records against the survey.
The first comes from the New York State Office of Temporary and Disability Assistance, which captures the monthly payments dolled out as part of the Supplementary Nutrition Assistance Program (SNAP, or food stamps), Temporary Assistance for Needy Families (TANF), and General Assistance for all people in New York state between 2007 through 2012. The second comes from the Department of Housing and Urban Development (HUD), which shows the rent paid by tenants.
Both are what Meyer calls "exceptionally accurate," and neither seem to line up with the story the CPS tells.
"We're talking about a huge gap," said Meyer, whose findings are in a recently published National Bureau of Economic Research paper. "When the numbers are corrected, we see that government programs have about twice the effect that we think they do."
Part of the problem is that many people who receive assistance today simply don't share that information. More than 60 percent of people who receive state-level general assistance, 40 percent of people who receive food stamps, and more than a third of people who receive housing assistance, didn't say so when surveyed by CPS.
But even when people do tell surveyors in person or by phone that the government is helping them, they don't do a very good job of relaying exactly how much they are getting. More than half of those who said they received food stamps underestimated the annual amount by more than $500. For public assistance, the portion of people who commit such errors was nearly 80 percent. For housing assistance, it was far worse.
Over the period Meyer observed, recipients underestimated how much they were given in the form of food stamps by 6 percent, general assistance by 40 percent, and housing assistance by nearly 75 percent.
Government agencies have used small adjustments to compensate for the erosion of the CPS, but much of it, Meyer says, is guesswork. "They never add up to the actual data," he said. "I can't think of an example where they fully compensated."
New York State is a good example. Between 2008 and 2012, the poverty rate was 13.6 percent before accounting for government assistance programs. By the survey's estimate, once food stamps, housing vouchers, and welfare were accounted for, the poverty rate dropped to 10.8 percent. Use the more accurate administrative data, however, and the combined effect of the programs brings the poverty rate down to 5.3%.
The truth is that surveys in general are becoming problematic. They are widely used in social science, and regularly relied upon in public policy, but they are fickle things. When people tell the truth and eagerly take part, as Americans did for many years, they tend to be wonderfully accurate. When people grow tired of answering questions, when they shy away from sharing truthful information about themselves, the gap between what surveys suggest and what is actually true begins to grow.
You can see hints of this problem in the following chart. That's true of most major household surveys. The chart below, plucked from a recent paper by Meyer, shows how the non-response rate has creeped up over the years for five key household surveys.
"We don't completely know why surveys have been getting so much worse, but we know that they're getting worse," said Meyer. "We know that for sure."
It's possible, Meyer says, that people are simply being over-surveyed these days. What used to be an exciting event—telling someone about your life—might just be an annoyance today. People are less likely to participate, and less willing to carefully respond when they do.
It might also be that the nature of modern surveying, which is very different than it was in the past, lends for less accurate information gathering. This certainly seems to be taking its toll on opinion surveys, which have also deteriorated in quality. Cellphones have complicated the process, which used to rely on calls to household phones, making it far more expensive. Cliff Zukin, a professor of public policy and political science at Rutgers University, explained the extent of the issue in a recent New York Times opinion piece:
To complete a 1,000-person survey, it’s not unusual to have to dial more than 20,000 random numbers, most of which do not go to actual working telephone numbers. Dialing manually for cellphones takes a great deal of paid interviewer time, and pollsters also compensate cellphone respondents with as much as $10 for their lost minutes.
No matter the reason, the fact that we depend on an increasingly faulty system to define things like poverty and judge programs that aim to alleviate it, is disconcerting. It can have a profound effect on policy discussions concerning the two.
On the one hand, it makes it look like the poor are doing much worse than they are. The official poverty rate now is higher it was three decades ago, but by almost any measure the poor are better off than they were then. Meyer believes that a more accurate gauge would show that things are better or, at the very least, not worse.
On the other, it does government assistance programs a great injustice, by making them appear less effective than they actually are.
"It makes us think that programs like food stamps aren't doing much," said Meyer. "It makes us think that they aren't working well, which in turn makes it much easier for conservatives to point their finger at them."