Challenge Index Challenged

"But I have not found such a school, and I think that the dropout and achievement gap factors are so closely tied to average family income that no inner city school would ever get close to the top 100 list of schools that did the best by these measures. Indeed, that would not be a measure of how good the schools were but how well-off their students' parents were. When we find a way that inner city schools can significantly reduce their dropout rates, as Garfield significantly increased its AP participation rate, then we have something worth measuring. Until then, you have a measure with no point. Low-income schools will lose your game every time."

Here are a few more points made by Rotherham and Mead, and my quick reactions:

The Challenge Index is wrong to make the number of graduating seniors such an important part of the calculation, since that means schools with high dropout rates will look better than they would otherwise.

That is exactly right, and it was one of the reasons why I wanted to use the number of graduating seniors as a measure of relative school size. I don't think there is anything the staff of an inner-city school can do at the moment with the tools available to them to increase substantially their number of graduates, with the possible exception of ignoring graduation standards, which I don't think anyone would favor. Given all their other disadvantages when being compared to suburban schools, I thought it fair to allow inner city educators to measure their AP and IB participation rates against that smaller core of students devoted enough to their studies to graduate, rather than the larger number of nominally enrolled students who just show up occasionally to see their friends.

A school with a small number of students taking many AP and IB tests can look good on the Challenge Index, even though most students are not involved in the program.

I have discovered that such small cabals of frantic college test-takers almost never do their schools much good on the list. With very few exceptions, the schools that do best on the Newsweek list are those that try hardest to involve as many students as possible in AP and IB. Most high schools let only relatively few of their students take AP or IB courses and tests, which is the major reason why the 1,069 schools that made last year's Newsweek list represent only about 4 percent of U.S. public high schools.

Newsweek should not use the word "best" in describing these schools, because so many of them have high dropout rates and large achievement gaps.

Best has become a very elastic term in America. Your list of best film directors may rely on their Oscar totals; mine may depend on their box office receipts. This inspires useful arguments, which is what Newsweek's list is doing for high schools. A list of the top schools using the criteria Rotherham and Mead suggest is going to be a list of the most affluent high schools. I don't think that is as good a use of the word "best" as a list of which schools are trying hardest to get as many students as possible ready for college by taking college-level exams, since the research shows that students who do well on those exams are more likely to graduate from college.

Newsweek should use more sophisticated measures.

The more sophisticated the measures, the more difficult it will be for readers to understand what the magazine is doing, and the more difficult it will be to make valid comparisons between individual schools, especially when they are in different states. Rotherham and Mead suggest -- somewhat apologetically, because they are not dumb -- that Newsweek might try to use some of the data that the states use to determine which schools have achieved adequate yearly progress under No Child Left Behind. The sad truth, however, is that these data points and the state decisions themselves are often beyond reader comprehension, and can be very subjective. Each state uses different tests and different standards of proficiency, so schools in Texas cannot be ranked against schools in Illinois. Rotherham and Mead also acknowledge that the dropout statistics we have at the moment are a mess.

Doing it their way would also eliminate one of the Challenge Index's great strengths -- it is so simple readers can understand it and do the arithmetic for their own neighborhood high schools. Rotherham and Mead seem to prefer the method U.S. News & World Report uses in its college rankings -- a complex assortment of several different data points weighted in ways whose validity only experts can judge. Rotherham and Mead are experts, and so they are comfortable with that, but the editors of Newsweek and I are journalists who think our job is to serve readers with as much clarity as possible.

Rotherham is one of the most active, influential and knowledgeable education experts in the country. I often quote him in my stories. He served in the Clinton White House as an education adviser and now sits on the Virginia Board of Education. If Mark R. Warner, the former Virginia governor who gave Rotherham that appointment, succeeds in his quest for the White House, I would not be surprised to see him make Rotherham U.S. secretary of education. Mead, who crunched most of the numbers in their report, is also a fine researcher who did great work with a very complex subject.

I am flattered two such people took the time and effort to critique the Challenge Index with such care, and I hope they won't stop there. One point they make repeatedly is that the Newsweek list, because of its focus on AP and IB, is overlooking similar schools that are doing a better job than those on the list in reducing dropout rates and closing achievement gaps.

Here is my request. Please find those schools, and if you do, tell me which ones they are. I don't believe they exist, but if they do, it will be an important discovery and well worth a few more columns. I don't think Rotherham and Mead will be able to find even one open-enrollment public high school in the United States that is not already on the full Newsweek list and has significantly better dropout and achievement gap results than schools with similar portions of disadvantaged students that they say should not be on the list. I think that high AP or IB participation is a key indicator of a school that is moving in the right direction on other measures of quality, so I will be very surprised if there is an inner city school that doesn't reach the Challenge Index mark but does better than those that are on the list on graduation and achievement rates.

But if they do find such schools, I will be very glad, and eager to help them and Newsweek and The Post come up with an entirely new way of rating high schools, for the betterment of all of us, especially our children.