Andy and Me: Two Ways to Rate High Schools

By Jay Mathews
Washington Post Staff Writer
Tuesday, December 4, 2007 12:28 PM

On Dec. 13, The Washington Post will mark the 10th year of the Challenge Index, my high school rating system, with our latest ranked list of all 185 public schools in the Washington area. Since 1998, Newsweek magazine also has been publishing its national best high schools list using the same method.

I am particularly excited this time because we have some competition. U.S. News & World Report, at the urging of Andrew J. Rotherham, my friendly adversary on this issue, has just published its own " America's Best High Schools" list at I have long celebrated what I call the School Rating Scoundrel's Club, composed of those of us who think that rating and ranking -- despite their many critics -- are useful ways to help readers figure out which schools are best for them. I admire the U.S. News college rankings and am intrigued by its new high school list. It is strengthened by Rotherham's commitment to improving schools, but it is also too complicated for its own good.

The Challenge Index rates and ranks schools by just one number, the college-level test participation rate, calculated by dividing the number of Advanced Placement, International Baccalaureate or Cambridge tests -- college-level exams given in high school -- by the number of graduating seniors. The U.S. News list mixes together several numbers. It looks for schools in the 40 states for which it has data whose average state test scores exceed statistical expectations and whose minority proficiency rates exceed state averages for those groups. Schools that survive that initial screening are then ranked based on a weighted formula that includes both AP test participation and AP test scores.

The essential differences between the two ways of ranking reflect the differences between Rotherham and me. Only 36, Rotherham has served as an education adviser to President Bill Clinton, has founded two education policy and research organizations and is a member of the Virginia Board of Education, the youngest appointee to that board in modern times. He is a policy maker. His high schools list is based on key factors in the policy process: test scores, minority achievement and college readiness as measured by AP participation and success. U.S. News and the statisticians at Standard & Poor's, led by Paul Gazzerro, the director of analytical criteria for School Evaluation Services, have compiled the list using a basic policy-making tool--data collected each year by state governments.

At 62, I have been a journalist for 40 years. With editors and researchers at The Post and Newsweek, I have relied on the basic reporting tools of direct contact with principals, teachers, students and parents. Our focus is not what works for policy makers but what is most useful for readers, particularly parents, trying to judge the quality of their local schools and others that might be available to them. I think college-level test participation is the only available comparative factor that allows parents to see how much value schools are adding to their children's lives. It is a much better measure than average test scores, which largely reflect not how good a school is but how wealthy and well-educated its parents are. AP, IB and Cambridge test participation also has the virtue of being a factor that smart school leaders can improve very quickly by removing the barriers to taking such courses that still exist in most high schools. Allowing all students to take AP and IB in turn creates incentives to improve preparation for those courses and tests and enrich the teaching of all classes. Raising the incomes and education levels of students' parents is not nearly so easy.

A look at the top 100 schools on Newsweek's list last May and on the U.S. News list just released shows three significant differences. The U.S. News list has eight of the most selective high schools in the country, with the Thomas Jefferson High School of Science and Technology in Northern Virginia at No. 1. The Newsweek list excludes such schools, placing them on a separate "Public Elites" list because the Challenge Index is designed to show which schools are challenging average students and doesn't work with public magnet or charter schools that have few or no average students. Secondly, several big urban schools with many low-income students and strong AP programs appear on the Newsweek list but not on the U.S. News list because of their low average state test scores. Third, about three dozen schools with strong IB programs are on the Newsweek list but not the U.S. News list, which only uses AP in its rankings.

I asked education experts with very diverse views what they thought of the new high school ranking rivalry. Those opposed to the test-based rating systems required by the federal No Child Left Behind law were, as you might guess, not happy with either magazine's list. Author Alfie Kohn, whose latest book is "The Homework Myth," said: "Just as many colleges are finally beginning to opt out of U.S. News & World Report's infamous rankings, pointing out the damage done by that list and the absurdity of its premises, the magazine decides to extend this nonsense to high schools. The whole enterprise is noxious; it's about winning, not learning--as though we were talking about football teams. And one suspects the point is to sell magazines, not to provide useful information.

"But even people who go in for this sort of thing ought to be appalled because the magazine's criteria for success are about standardized tests, which measure what matters least. A school with rising, or unexpectedly high, scores often has become little more than a glorified test-prep center, sacrificing recess, project-based learning, the arts and a host of other things that help kids to become enthusiastic and proficient learners. This list not only confuses meaningful learning with high test scores; it actively contributes to a culture where schools give up the former to attain the latter so they can become--God help us--'winners.' "

Inner-city educator and author Deborah Meier said she has been drawn to what she calls "the Consumer Reports model" as an alternative to the Newsweek and U.S. News rankings. The Consumer Reports people, she said, "do occasionally rate someone as 'best buy'--but mostly they just provide information on a wide range of stuff. . . . Mostly, it's not comparative--at least overall. We are judging apples vs. oranges. I'm interested in wagons that are fairly small, with four-wheel drive and good gas mileage. Then I get a chance, after all, to drive it before I buy it and look for little things that matter to me--the size of the numbers on the mileage meter, the location of certain interior lights . . . The ranking game, the constant urge to compare everything--who's better, etc.--is a bad and injurious habit. I have three kids and I truly can't think of any way of ranking them that makes sense--and surely not over time!"

Educational psychologist and author Gerald W. Bracey gave a brief friendly nod to Challenge Index. He said it "is more 'democratic' because it rewards those who encourage participation, not those who score high. Obviously, there are few schools in the country which can compete with Thomas Jefferson on achievement measures."

But the heart of Bracey's argument: "I don't think we should be ranking the schools at all. In part, I just think it's unhealthy. In part, both methods have such a narrow focus. It's like ranking an athlete or a team on a single measure. Yes, they are ranked, but it's a competitive arena. It's like ranking the Key School in Indianapolis on test scores. The school is formed around Howard Gardner's theories of multiple intelligences, and many of the things they do each day have nothing to do with test scores."

Four other experts who more or less support the national effort to raise school achievement measured by test scores saw some advantages, but also drawbacks, in high school rankings. Mike Riley, the outgoing Bellevue, Wash., school superintendent, who is soon to join a College Board effort to raise low-income student achievement, said what he liked about the new U.S. News list is "it tries to consider other factors. A truly comprehensive view of schools would be very helpful, not only for the purposes of a rating system but also for its value in expanding the way everyone--the public, parents, educators--think about what school 'success' means."

Riley had a problem, however, with the U.S. News screening method. "If a school meets expectations on state testing--but doesn't exceed expectations on this measure--it fails to pass through the gate and the number of kids who pass a nationally recognized test (AP) is never considered. I wonder if they have this backwards. A national measure puts all schools on the same playing field while state tests are wildly uneven in terms of their difficulty levels and knowledge and skills measured."

Gerald N. Tirozzi, executive director of the National Association of Secondary School Principals, said his organization "is not a proponent of school rankings. We recognize that the schools included on both the U.S. News and World Report and Newsweek Challenge Index lists are deserving of the honors they have received. But how can we truly compare one school to another when each state has its own set of standards? Until we have national standards, at the very least in the areas of reading and math, rankings of this type do not hold nearly as much weight as they could if all schools were on a level playing field."

Mel Riddile, a former National High School Principal of the Year who now runs T.C. Williams High School in Alexandria, objected to the top U.S. News ranking for Jefferson. "Comparing schools that can improve test scores by sorting applications does nothing for schools like T.J. because they know they are good. They should be. They are the all-star team. U.S. News is terribly naive to construct a rating that compares results and fails to take into account inputs and processes. This rating, while it seeks to find the schools doing the best, only identifies schools that have the best. This ranking system is like comparing the American League All-Star team to that of the Baltimore Orioles who have not had a winning season for a decade and saying that the all-star team is statistically superior. This type of ranking does a disservice to efforts to reform or reinvent American high schools, because it says to school boards, communities, principals and teachers in diverse, open enrollment schools, that you can never be good unless you stack the deck. . . . The best schools are those that do the most with the students, all the students, who walk through their doors, not the best qualified applicants."

Dick Reed, a very active high school parent in Fairfax County, who has studied the AP and IB programs, said the U.S. News rankings were "a good list, though I have quibbles," mostly having to do with the complex format. "It would be almost impossible for an interested person to do their own comparative scoring. If all that's wanted are the top schools, it's fine. If I wanted to get my school in this list somehow, I would have almost no idea how to do so. That's not true with the Challenge Index. To get higher on the Challenge Index, I simply press my school to encourage more students to take IB or AP classes. Nothing more need be done--nothing more can be done. With the new list, there's far more involved and almost none of those things can be affected by an individual."

Reed endorsed, however, the idea behind both lists: "People are naturally competitive. Harness that instinct and use it to raise the performance of a school."

There are limits to the usefulness of this competitive instinct, of course. To see it taken to its hilarious extremes, check out the high schools the Wall Street Journal ranked last Friday as the most likely to get your child into eight selective colleges, including Harvard and Princeton. They were almost all very expensive private schools, or very selective public ones, including Jefferson (roughly 36th out of 40 schools.) The accompanying articles mentioned that these schools have counselors who are experts on gaming the admissions process, but did not note another explanation for their success: They likely have the largest percentage of parents who are alumni of the targeted colleges, thus getting extra legacy points for their kids.

I hope there will be many more discussions of what really makes a good high school. As a U.S. News statement explained, the new list was inspired in part by an article Rotherham and research colleague Sara Mead wrote saying it was wrong for the Newsweek list to include large urban schools that had strong AP or IB programs but low average scores on state tests and high dropout rates. They let me respond on Rotherham's Web site, I said those schools most needed and deserved the encouragement of recognition in a national list. The Challenge Index grew from my five years of writing a book about Garfield High School in East Los Angeles, where AP teachers had changed hundreds of lives despite being in a very large urban school with poor state test scores because of the steady influx of low-income Hispanic students. Garfield proved those students could perform very well on AP exams if given extra time and encouragement. The list was designed in part to show other such inner-city schools they could do the same for many of their students if they followed Garfield's example.

Rotherham argued that Garfield's achievements were admirable, but until inner-city educators found a way to raise state test scores and lower dropout rates for all students, they did not deserve to be recognized on a national list. I said those were diseases for which there was yet no cure, and those schools should not be denied recognition because they have not solved problems that remain insoluble.

There are some possible solutions, of course, on which both Rotherham and I agree. We have on each of our lists a few inner-city high schools like YES in Houston and Preuss in San Diego, which have shown what small charter schools intensely focused on raising low-income student achievement can do. My hope is that both the Newsweek and U.S. News lists will evolve toward our shared goal -- improving education for the many American children who are not getting the teaching they deserve. We are both searching for schools to celebrate, but in different ways. The U.S. News list reminds me of a big government operation, with satellite data and computers, tracking schools from 200 miles up. Those of us working on the Challenge Index at Newsweek and The Post are on the ground, peeking into schools, calling or faxing or e-mailing principals, teachers and counselors, collecting their data, discussing their initiatives and asking them for their reactions to what we are doing.

Rotherham, Gazzerro and the fine journalists at U.S. News are going to discover that when they don't put a high-achieving school on their list, many people will inquire about that. Alumni of Benjamin Banneker Academic High School in the District--which along with 10 states was unable to provide data to U.S. News--are already complaining about their school going unmentioned. Fairfax County, the largest school district in Rotherham's state, Virginia, has all 25 high schools (not counting Jefferson) on the Newsweek list of 1,350 schools, the top 5 percent in the United States, but only five, including Jefferson, on the U.S. News list. I suspect he will be hearing from those left out, all part of the learning process for us school-rating scoundrels.

Join me for a live chat online today at 2 p.m. at

View all comments that have been posted about this article.

© 2007 The Washington Post Company