Before Hurricane Katrina, one of New Orleans’ most systemic ills was its dysfunctional school system. After the storm, officials did something unprecedented — they started from scratch. The state took over almost all district schools and turned them over to nonprofit charter management organizations. These new charter leaders made major changes in the teacher workforce. The district’s teachers were all fired. The collective bargaining agreement was ended. Schools hired more young teachers from outside the city, prepared in alternative preparation programs, such as Teach for America.

Did it work? On this question, the national media has been all over the map. A New York Times op-ed concluded, “it is wiser to invest in improving existing education systems than [as in New Orleans] to start from scratch,” and that New Orleans-style reforms have “hurt the most disadvantaged students.” Malcolm Gladwell wrote in the New Yorker that the New Orleans school reforms “increased test scores far less than hoped.” But an article in New York Magazine concluded that the school reform in New Orleans, “is the breakthrough in social equity liberals have been waiting for.” A similar message came from the Washington Monthly and CNN anchor Campbell Brown’s pro-reform organization.

Given the importance of the New Orleans experience, a team of 20 national researchers and I spent more than a year carrying out a dozen separate studies on New Orleans schools. From this, it is clear the national media portrayals are missing the point.

Before Katrina, the school system was corrupt and dysfunctional, running through a new superintendent every 11 months. The FBI had so many investigations going on that it opened an office within the school district. Third-party evaluations of district operations identified a litany of problems. Student test scores were among the worst in the state, and the country. Change from within an existing system might work in some cases, but the existing education system had failed in New Orleans.

In a study I conducted with Matthew Larsen, we found that the city’s test scores rose dramatically because of the post-Katrina reforms. Even the most pessimistic estimates suggest that the reforms significantly increased scores (and probably high school graduation rates and college entry) and more than alternative policies and programs would have. These achievement gains also occurred across the board. In this respect, low-income students were not hurt. They benefited academically.

That being said, some of the rhetoric of reform supporters has gone overboard. There are some real issues and questions, just not the ones that these critics have set their sights on.

For example, though disadvantaged students benefited, they seem to have benefited less than other groups. Early on, as this entirely new type of system was being put in place, there were real horror stories about how special education students and others were suspended and expelled at high rates. Under pressure from community groups, state and local leaders took several steps to address the problem, yet it remains unclear whether the problems are solved.

Critics are concerned that schools under the reforms are too focused on test scores. This is a national concern as well, but the intensity of test-based accountability in New Orleans is even stronger and may reduce focus on other important educational goals like creativity and local cultural knowledge. In the coming years, we’ll get a better sense of the real results by looking at college and beyond.

One potential weakness of a system of autonomous schools like the New Orleans model is that disadvantaged students can more easily fall between the cracks. With neighborhood attendance zones, a specific school is responsible for each student. With school choice, tens of thousands of students are in the hands of one or two district staff people. And there are signs that high school dropouts are being under-reported.

Finally, whatever lessons we might draw from New Orleans may be exclusive to New Orleans. Our student outcomes had nowhere to go but up. New Orleans also saw a massive influx of federal and philanthropic funding and skilled people from across the country that other cities are unlikely to experience. Other districts should look to New Orleans, but tread carefully.

Hyperbole is no basis for school improvement and no way to honor Katrina’s victims. Educators and policymakers around the country need to know how and how well the unprecedented New Orleans school reforms really worked. The people of New Orleans deserve to know. The evidence provides a lot of good news about the New Orleans school reforms, but school improvement is always more complicated than it seems.