I read with interest the new report by the National Research Council, an arm of the National Academies, “Incentives and Test-based Accountability in Education.” It summarized the results so far of using publicly reported test results to inspire and inform efforts to improve student achievement. As other reports have revealed, we have had some modest gains, but nothing to justify nailing a “Mission Accomplished” banner over the schoolhouse door.

I was bothered, however, by the way the expert NRC panel judged these results. “Test-based incentive programs, as designed and implemented in the programs that have been carefully studied, have not increased student achievement enough to bring the United States close to the levels of the highest achieving countries.”

Uh ... well ... sure. But is that a useful measure of progress? We haven’t brought lasting peace to the Middle East or democracy to China, but life has improved for many people in those regions. Is there something we should be doing that WILL overcome the cultural and political obstacles that keep our kids from outscoring the Koreans and the Finns? The report doesn’t say.

That is as far as I got in my analysis: interesting report, odd conclusion. Fortunately, Stanford economist Eric A. Hanushek has posted a more nuanced critique on the educationnext.org Web site.

He quotes the report’s we-gotta-catch-the-world statement, as well as the first sentence of the official news release: “Despite being used for several decades, test-based incentives have not consistently generated positive effects on student achievement.” His own reading of the report’s data is that they provide “a fairly thin evidentiary base but one that generally suggests positive impacts of accountability,” even if they are not nearly enough to make us number one.

Hanushek, whose specialty is education, says: “The biggest problem with the conclusion (and the report) lies in the casual ‘compared to what’ measure that is adopted. Why should we discard an effective program just because it falls short of our hopes of producing the world’s best education? ... Nowhere does the report indicate an alternative educational program that leads to as large an improvement in overall U.S. achievement as accountability. Nowhere does the report suggest any single program or package of reforms that would close the achievement gap with the highest performing countries. Nowhere does the report really make the case that alternative reform packages should not include an accountability component.”

Hanushek has done much work comparing various countries’ educational improvement to their economic growth. I don’t entirely trust that line of analysis, since the Japanese economy tanked despite fine school achievement results. But Hanushek’s data cover many countries. He is bothered that the NRC report “speaks quite dismissively of the estimates of achievement gains of 0.08 standard deviations.” Based on his research with Ludger Woessman, Hanushek says, that small improvement in school achievement could mean trillions of dollars more in the gross domestic production.

It might have helped, Hanushek says, if the report suggested what we might do to improve the system we have. Americans are tinkerers, and do better if we work with materials at hand

Some pundits are suggesting that the NRC report is a big step forward for America, pulling us back from disaster. I don’t see that. Show me something that works better and I will go for it. Discarding the system we have out of frustration that it hasn’t made our schools the best is a recipe for confusion, and even less progress.