With a few exceptions, however, I can’t really do that — at least not with the precision my health colleagues often can. More important, neither can policymakers in Congress and in many regulatory agencies. The Food and Drug Administration has more information available in deciding whether to approve a treatment that a few thousand people will receive than Congress does in considering a bill that will affect every American.
Each year, hundreds of carefully controlled, double-blind studies are done to learn whether a given pill is better than a placebo or whether a new surgery leads to quicker recoveries. Many of these studies are funded by a single agency, the National Institutes of Health. By contrast, in a typical year, no such studies are conducted to evaluate social policy proposals.
That’s not because such studies are impossible. In 1962, researchers in a small Michigan school district randomly selected 58 3- and 4-year-olds to enroll in a preschool program, then spent decades comparing them with a control group of 65 kids who didn’t go to preschool. Those who enrolled learned more and made more money as adults. In 1976, the Chicago Housing Authority randomly placed public-housing residents in apartments either in the city or in the suburbs, and then tracked the two groups. Those given places in the suburbs did better on every metric from household income to their children’s rates of college attendance.
Those studies had a big impact. The Chicago study, for example, is the main research cited in proposals to provide housing vouchers to poor families to break up pockets of “concentrated disadvantage.”
But studies like those are very rare. They’re expensive, which discourages universities and school districts (such as the one in Michigan conducting the preschool study) from doing them. And often, as in the Chicago case, they come about only because a court orders them
Because of this rarity, it’s easy to pick nits. The preschool study involved only children with low IQs, critics noted. Maybe the results would have been different with children of average or above-average intelligence. The Chicago housing study was conducted during a time when crime was much worse than today. Maybe with safer inner cities, you wouldn’t see similar gains from sending families to the suburbs.
Researchers have spent gallons of ink arguing over such caveats, trying to figure out what can and can’t be inferred from the meager pool of good data with which they’re forced to work. At no point does the straightforward solution present itself: run another study.