A new study on the effectiveness of public charter schools concludes that most of the research on the subject has been conducted with methods that “tell us little about causal effects.”

The study, just published in the journal Science, was conducted by two well-known researchers: Julian Betts, an educational economist at the University of California at San Diego and executive director of the San Diego Education Research Alliance, and Richard Atkinson, president emeritus of the University of California, former director of the National Science Foundation and professor emeritus of cognitive science and psychology at UC San Diego.

They wrote in their study, “Better Research Needed on the Impact of Charter Schools,” that charter schools have been embraced by the Obama administration — and by the George W. Bush and Clinton administrations before it — as “the saviors of a broken educational system.” But, they said, researchers still can’t answer the question: Does attendance at a charter school improve student outcomes?

“Which charter schools, or even types of charter schools, are more effective than others? We don’t really know,” they wrote.

Researchers (one of them was Betts) conducting a recent meta-analysis of charter research studies ended up throwing out about 75 percent of them because they didn’t take into account differences between the backgrounds and academic histories of students attending charter schools and those attending traditional public schools.

“Most studies simply take a snapshot of student performance at a single point in time,” the new report says. “Such studies cannot disentangle school quality from the preexisting achievement level and trajectory of students who decide to attend a given school. The potential for student self-selection into charter schools is great, which makes naïve comparison of student outcomes at charter schools and traditional public schools misleading.”

According to Betts and Atkinson, some of the most rigorous research on charters has been done on children who participate in admission lotteries held by popular schools that are oversubscribed. Both winners and losers in these lotteries are considered “an ideal control group,” they said, so comparing outcomes is “the closest we can get to a randomized controlled experiment.”

Lottery-based studies suggest that charters do as well as or better than traditional public schools, they wrote. But here’s the problem with that thinking: Those studies have analyzed only about 2 percent of the charter schools nationally. In fact, most charter schools don’t hold admissions lotteries, and it is reasonable to assume that those that do are perceived by parents as being especially successful.

Betts and Atkinson write that “value-added” research would be useful for schools that don’t have lotteries but that it is difficult to get the student test data required to do the job correctly.

They argue that more lottery-based studies are needed as well as more longitudinal work, which means that researchers should be able to routinely access student data — and obtain lottery data from charter schools.

Until better research is done, policymakers simply won’t have the appropriate data to make decisions about which charters to replicate and which to shut down, Betts said in a university press release.


Follow The Answer Sheet every day by bookmarking http://www.washingtonpost.com/blogs/answer-sheet.