Andy and Me: Two Ways to Rate High Schools
Tuesday, December 4, 2007; 12:28 PM
On Dec. 13, The Washington Post will mark the 10th year of the Challenge Index, my high school rating system, with our latest ranked list of all 185 public schools in the Washington area. Since 1998, Newsweek magazine also has been publishing its national best high schools list using the same method.
I am particularly excited this time because we have some competition. U.S. News & World Report, at the urging of Andrew J. Rotherham, my friendly adversary on this issue, has just published its own " America's Best High Schools" list at usnews.com. I have long celebrated what I call the School Rating Scoundrel's Club, composed of those of us who think that rating and ranking -- despite their many critics -- are useful ways to help readers figure out which schools are best for them. I admire the U.S. News college rankings and am intrigued by its new high school list. It is strengthened by Rotherham's commitment to improving schools, but it is also too complicated for its own good.
The Challenge Index rates and ranks schools by just one number, the college-level test participation rate, calculated by dividing the number of Advanced Placement, International Baccalaureate or Cambridge tests -- college-level exams given in high school -- by the number of graduating seniors. The U.S. News list mixes together several numbers. It looks for schools in the 40 states for which it has data whose average state test scores exceed statistical expectations and whose minority proficiency rates exceed state averages for those groups. Schools that survive that initial screening are then ranked based on a weighted formula that includes both AP test participation and AP test scores.
The essential differences between the two ways of ranking reflect the differences between Rotherham and me. Only 36, Rotherham has served as an education adviser to President Bill Clinton, has founded two education policy and research organizations and is a member of the Virginia Board of Education, the youngest appointee to that board in modern times. He is a policy maker. His high schools list is based on key factors in the policy process: test scores, minority achievement and college readiness as measured by AP participation and success. U.S. News and the statisticians at Standard & Poor's, led by Paul Gazzerro, the director of analytical criteria for School Evaluation Services, have compiled the list using a basic policy-making tool--data collected each year by state governments.
At 62, I have been a journalist for 40 years. With editors and researchers at The Post and Newsweek, I have relied on the basic reporting tools of direct contact with principals, teachers, students and parents. Our focus is not what works for policy makers but what is most useful for readers, particularly parents, trying to judge the quality of their local schools and others that might be available to them. I think college-level test participation is the only available comparative factor that allows parents to see how much value schools are adding to their children's lives. It is a much better measure than average test scores, which largely reflect not how good a school is but how wealthy and well-educated its parents are. AP, IB and Cambridge test participation also has the virtue of being a factor that smart school leaders can improve very quickly by removing the barriers to taking such courses that still exist in most high schools. Allowing all students to take AP and IB in turn creates incentives to improve preparation for those courses and tests and enrich the teaching of all classes. Raising the incomes and education levels of students' parents is not nearly so easy.
A look at the top 100 schools on Newsweek's list last May and on the U.S. News list just released shows three significant differences. The U.S. News list has eight of the most selective high schools in the country, with the Thomas Jefferson High School of Science and Technology in Northern Virginia at No. 1. The Newsweek list excludes such schools, placing them on a separate "Public Elites" list because the Challenge Index is designed to show which schools are challenging average students and doesn't work with public magnet or charter schools that have few or no average students. Secondly, several big urban schools with many low-income students and strong AP programs appear on the Newsweek list but not on the U.S. News list because of their low average state test scores. Third, about three dozen schools with strong IB programs are on the Newsweek list but not the U.S. News list, which only uses AP in its rankings.
I asked education experts with very diverse views what they thought of the new high school ranking rivalry. Those opposed to the test-based rating systems required by the federal No Child Left Behind law were, as you might guess, not happy with either magazine's list. Author Alfie Kohn, whose latest book is "The Homework Myth," said: "Just as many colleges are finally beginning to opt out of U.S. News & World Report's infamous rankings, pointing out the damage done by that list and the absurdity of its premises, the magazine decides to extend this nonsense to high schools. The whole enterprise is noxious; it's about winning, not learning--as though we were talking about football teams. And one suspects the point is to sell magazines, not to provide useful information.
"But even people who go in for this sort of thing ought to be appalled because the magazine's criteria for success are about standardized tests, which measure what matters least. A school with rising, or unexpectedly high, scores often has become little more than a glorified test-prep center, sacrificing recess, project-based learning, the arts and a host of other things that help kids to become enthusiastic and proficient learners. This list not only confuses meaningful learning with high test scores; it actively contributes to a culture where schools give up the former to attain the latter so they can become--God help us--'winners.' "
Inner-city educator and author Deborah Meier said she has been drawn to what she calls "the Consumer Reports model" as an alternative to the Newsweek and U.S. News rankings. The Consumer Reports people, she said, "do occasionally rate someone as 'best buy'--but mostly they just provide information on a wide range of stuff. . . . Mostly, it's not comparative--at least overall. We are judging apples vs. oranges. I'm interested in wagons that are fairly small, with four-wheel drive and good gas mileage. Then I get a chance, after all, to drive it before I buy it and look for little things that matter to me--the size of the numbers on the mileage meter, the location of certain interior lights . . . The ranking game, the constant urge to compare everything--who's better, etc.--is a bad and injurious habit. I have three kids and I truly can't think of any way of ranking them that makes sense--and surely not over time!"
Educational psychologist and author Gerald W. Bracey gave a brief friendly nod to Challenge Index. He said it "is more 'democratic' because it rewards those who encourage participation, not those who score high. Obviously, there are few schools in the country which can compete with Thomas Jefferson on achievement measures."
But the heart of Bracey's argument: "I don't think we should be ranking the schools at all. In part, I just think it's unhealthy. In part, both methods have such a narrow focus. It's like ranking an athlete or a team on a single measure. Yes, they are ranked, but it's a competitive arena. It's like ranking the Key School in Indianapolis on test scores. The school is formed around Howard Gardner's theories of multiple intelligences, and many of the things they do each day have nothing to do with test scores."
Four other experts who more or less support the national effort to raise school achievement measured by test scores saw some advantages, but also drawbacks, in high school rankings. Mike Riley, the outgoing Bellevue, Wash., school superintendent, who is soon to join a College Board effort to raise low-income student achievement, said what he liked about the new U.S. News list is "it tries to consider other factors. A truly comprehensive view of schools would be very helpful, not only for the purposes of a rating system but also for its value in expanding the way everyone--the public, parents, educators--think about what school 'success' means."