Challenge Index Challenged

Newsweek magazine plans to publish in a couple of months another edition of its "America's Best High Schools" list, based on a rating system called the Challenge Index that I conceived a decade ago. It is a very odd and narrow measuring device, and even though I think it is the best single statistical measure of public high schools, it takes a while to understand why that is. Thus the most common criticisms of it can be summed up in one word: "Huh?"

That is changing, as more people become accustomed to the idea of ranking schools based on participation in college-level tests and more experts judge the Newsweek list -- and a similar list of Washington-area schools in The Washington Post -- on what the Challenge Index tries to measure, rather the many important things about high schools that nobody can yet measure adequately, such as teacher quality and student motivation.

The most useful contribution to this debate so far is a 19-page article posted last month on www.educationsector.org by Andrew J. Rotherham, co-founder and co-director of the new Washington-based independent education think tank, Education Sector, and Sara Mead, the organization's senior policy analyst. Rotherham and Mead conclude that the 100 top schools identified by Newsweek cannot be accurately called the nation's best because many of them have very high drop-out rates and large gaps in test scores between white and minority students. They suggest Newsweek change the title of the list and use some of the test-score and drop-out data coming out of the No Child Left Behind Act to rate schools in different and more complex ways.

What I like best about the Education Sector piece, "Challenged Index: Why Newsweek's List of America's 100 Best High Schools Doesn't Make The Grade," is that it discusses other ways to rate high schools that may help improve learning for all students. The Challenge Index was born out of my frustration with the widespread notion that the best high schools are the ones with the fewest low-income and minority students, and the worst are theones with the most students in those categories. I sense Rotherham and Mead agree with me that such an assessment is not right, and that schools should be judged not on their average family-incomes but on how firmly and successfully each school works to raise achievement. Rotherham says he and Michael Goldstein, founder of the Media and Technology Charter High School, MATCH, in Boston, are working on their own top 100 list using the ideas in his article, and I look forward to that.

But I don't agree with the changes Rotherham and Mead suggest Newsweek make in the way it rates high schools because I think, despite the authors' good intentions, their ideas will lead to a list that recognizes only those schools in the nicest neighborhoods with the greenest lawns, the most educated parents and the fewest disadvantaged kids.

The Challenge Index rates schools by a very simple formula. It takes the number of Advanced Placement and International Baccalaureate tests taken by all students in May and divides that total by the number of seniors graduating from the school in June. The Newsweek list is actually much larger than the 100 schools that are ranked in the magazine. Last year, on the magazine's and The Post's Web sites the Newsweek researchers and I identified 1,069 schools that achieved a rating of at least 1.000, including all the public schools that had at least as many AP or IB tests as graduating seniors.

Rotherham and Mead acknowledge in their article, studded with charts, that "taking AP and IB tests is one key indicator of a good high school," but say "we believe that the Challenge Index is a seriously flawed measure of overall quality."

"A successful high school should show high levels of student achievement, graduate almost all of its students and not let any demographic subgroup suffer at the expense of others," they say. "Most national and local experts and policymakers share these values. To be sure, graduation rates and student achievement are hardly the only indicators of a school's quality. At a minimum, however, America's best high schools should be expected to meet these basic criteria.

"Yet our analysis shows that many schools on Newsweek's list do not meet these minimum standards. Using publicly available student performance data, we found that many schools on Newsweek's 2005 ranking have glaring achievement gaps and high dropout rates. By presenting them as America's best, Newsweek is misleading readers and slighting other schools that may in fact be better than those on Mathews' list. For example, the magazine ranks Eastside High School in Gainesville, Fla., as the third best high school in the nation, but only 12 percent of Eastside's black students were reading at grade-level in 2004. And Newsweek ranks Hillsborough High School in Tampa, Fla., as America's 10th best high school, but only 17 percent of black students and 26 percent of Hispanic students met the state's modest grade-level standards in 2004. While some students at Eastside or Hillsborough may be receiving a challenging education, it's clear that many are not. And Eastside and Hillsborough are not outliers. In fact, schools with substantial inequities in student achievement make up a significant proportion of Newsweek's list of best high schools."

What Newsweek should do instead, Rotherham and Mead say, is broaden its measuring system to include much more data, not just AP and IB participation rates. The magazine's editors, they say, should use "disaggregated data and graduation rates in their rankings and reward schools that do well with students from diverse backgrounds. As state data systems become more developed, information about course-taking patterns and other measures of college preparation -- as well as longitudinal data about performance -- could also be included."

Rotherham and Mead kindly showed me an advance copy of their article and included my lengthy response on their Web site. My principal complaint was that they were ruling out any recognition in the Newsweek list of several inner city schools that have made substantial improvements in the way they prepare students for college, but have yet to solve the achievement gap and drop-out problems. This is especially galling to me, I said, because one of the schools they would ban from the list with their new criteria is Garfield High in East Los Angeles, where I first learned that low-income students could achieve at a very high level if given enough time and encouragement to do so.

"There are a few inner city schools like Garfield that have produced unusually encouraging and resourceful teachers and impressive AP and IB participation rates," I said in my response to Rotherham and Mead, "but because the vast majority of their students are low-income, they have not made much progress yet on the dropout and achievement gap problems you properly identify. If I knew of any inner city public school with a majority of low-income students that had a significantly lower dropout rate and achievement gap than other schools with similar demographics, I would write about it and then follow your suggestion and look for a way to measure its achievement and rank other schools accordingly.