By Jay Mathews
Washington Post Staff Writer
Tuesday, August 1, 2006 11:58 AM

From the very beginning I knew that publishing a ranked list of public high schools, based on participation in college-level tests, would get me into trouble. I wasn't breaking any laws and I figured my critics, for the most part, would be very polite, since that is the way people interested in education usually are. But I knew I wasn't going to do my reputation any good. Here is the first paragraph of my introduction to my first high school list, the 1998 Challenge Index at the end of my book "Class Struggle":

"Nearly every professional educator will tell you that ranking schools is counterproductive, unscientific, hurtful, and wrong. Every likely criterion you might use in such an evaluation is going to be narrow and distorted. A school that stumbles one year may be fine the next."

And yet, I said, as a reporter and as a parent, I thought in some circumstances such ratings could be useful. In last week's column , high school social studies teacher Mark Crockett and I argued about this. Like many people, Crockett does not think that rating schools by Advanced Placement or International Baccalaureate tests in Newsweek and the Washington Post makes much sense. That discussion will continue, if the continual flood of e-mails questioning my sanity is any indication.

But it is important to note, I think, that I am no longer the only person doing this kind of comparative high school assessment. There are enough of us to form what we might call the School Rating Scoundrels Club. Fortunately, many of the new members are a lot smarter than I am.

They include Nicholas Colangelo, director of the Belin-Blank Center for Gifted Education and Talent Development at the University of Iowa; Mike Reno, a trustee of the Rochester (Mich.) Community Schools, and Tim Norris, personnel director of the Mountain Brook (Ala.) schools. There are also the several publications, including the Seattle Times, Westchester Magazine, Washingtonian magazine and others, that have used approximations of the Challenge Index in their own high school lists.

Colangelo has just published his second annual Iowa AP Index, to date the most ambitious effort I know of to apply the Challenge Index rating system to all high schools, public and private, in an entire state. His center's report can be found at http://www.iowaapindex.org/.

Colangelo said he thought paying close attention to each school's AP DATA would be a good way to encourage Iowa schools to be more challenging. Only four Iowa public schools qualified for the latest Newsweek list of the country's most challenging public high schools. Colangelo discovered that of the 389 public and private high schools in Iowa, only 213 had at least one student take an AP exam in 2005. Of that group, 187 schools--171 public and 16 private--consented to participate in the Iowa AP Index. "The top 25 schools range in class size from 11 to 378," said the report, co-authored by Colangelo, Susan Assouline, Damien Ihrig and Clar Baldus. "There are 20 public and 5 private schools in the top 25. The #1 school is Rivermont Collegiate High School, a small private school in Bettendorf. The biggest school (based on graduating seniors) in the top 25 is Iowa City High School in Iowa City, Iowa [378 seniors]. The smallest school is Russell High School in Russell, Iowa [11 seniors]." The University of Iowa researchers even found an Iowa school, Roosevelt High in Des Moines, that qualified for the Newsweek list but that I had missed, a mistake I plan to rectify soon.

Mike Reno, the school board trustee in Rochester, Mich., embraces Colangelo's view that more attention should be paid to individual school AP data. (Neither of them has tried to look at IB data yet, but I suspect that will come.) Reno said he suspected AP participation in his wealthy county was poor, despite claims from the school administration that it was tops in the state. So he started collecting his own data, resulting in a remarkable 41-page report, "Advanced Placement Participation in Michigan," available at his website, http://www.rcs-reno.com/.

Reno ranked 94 Michigan school districts by AP test participation. He discovered that Rochester was only number 24. Among 51 states and the District [see the list of states near the top of this column], Michigan ranked only 29th. The ratings, Reno concluded, were influenced as much by the emphasis educators and state officials placed on challenging courses as they were by parental income. "Many states demonstrate their commitment to AP in a very public way," he said, "such as tracking and publishing AP statistics on their websites and offering financial incentives to teachers and schools."

He concluded that "many Michigan children are at a disadvantage due to the concentration of AP participation in a relatively few number of schools. Over two thirds of AP exams are taken in districts representing just one-third of Michigan students. One-third of AP exams are taken in schools representing just seven percent of Michigan high school students." Reno invented his own measuring device, the Balanced Achievement Indicator, to show which schools are both bringing their students up to the minimum achievement levels required by the state and providing the higher academic challenge of AP.

At Mountain Brook, Norris is exploring the connection between AP participation and average scores on the SAT and ACT tests. He has gathered data on individual schools in many other states that share Mountain Brook High School's high performance levels.

The number of school rating scoundrels is growing. They probe and analyze and try to share what they find. I think they deserve a bigger audience. Nothing convinced me of this more than a letter from a very astute judge of high schools, educational consultant Jan Rooker, that was sent recently to the principal of New Canaan (Conn.) High School and the New Canaan school board. She was reacting to a story in the New Canaan News-Review quoting principal Tony Pavia as being skeptical of all such high school lists. He acknowledged he had not looked at the Newsweek list, which ranks New Canaan High at 911, in the top 4 percent of all U.S. schools but perhaps lower than what that very affluent community might expect.

Rooker started by saying that New Canaan is "an excellent high school" and I was wrong to base my high school evaluation "solely on one thing." But, she added, I was not the only one who, at least in part, appraises the worth of a high school based on AP participation.

Rooker advises students on college admissions. She said she has asked 85 college admissions officers in the past two years what was the first thing they look for in applicants' transcripts. She said each told her it was "the level of difficulty of the courses taken by a student. It is an automatic assumption that if an able student does not take AP courses when his or her high school offers them, then he or she has chosen not to challenge him or herself."

On that scale, she said, "some New Canaan students appear to be slouches. Why? Because even though NCHS offers AP courses, not many NCHS students take them. Such students appear simply to have chosen not to challenge themselves."

"Yet, I have worked with many students from NCHS who have wanted to take AP courses. They are told, though, that this is not a suitable placement for them. They hear things like: their past performance is not good enough, or they haven't been in honors level classes for the previous grade, or there is only one class and it is full."

Rooker said this is especially true of students with scores between 1250 and 1450 on the old SAT scale of 1600. She said she works with students from several other Connecticut high schools, and sometimes finds students with 1200 on their SATs taking four or five AP courses while New Canaan seniors with 1400s take none. "This is not because they are lazy, but because they were prevented from taking AP classes at NCHS by the system of placement." She said some NCHS staffers say the colleges its students apply to know that the school's non-AP courses are very rigorous, but that argument doesn't work if a New Canaan student applies to a college outside New England that doesn't know much about the school.

"My question is this: in a wonderful school like NCHS where there are a plethora of capable students and capable teachers, why are all able students not offered the opportunity to take AP courses?" Rooker said. "If this means changing the selection criteria, then that should be done. If it means adding more sections of some courses, then that should be done. If it means adding more AP courses, then that should be done. Why are many able students left to appeal and petition for placement in an AP courses, hoping to be admitted, only to find out that they aren't? . . . Our students are caught in the trap of not being allowed to take AP courses by their local high school, and then are being judged by college admissions committees for not taking them."

David Abbey, the New Canaan schools superintendent, said "we feel we have a strong and comprehensive AP program, but we don't open enroll and we don't require that those students who participate in the AP program take the AP tests." He said he felt that the Newsweek list penalizes schools like his that may have many students in AP classes, but not as many taking the AP tests.

He said he thought the most important consideration was how well a school was challenging all of its students, not how they were using AP "as leverage to get into college." He said the criteria for admittance to the school's AP courses were clearly listed on the school website and course catalog, and that the administration and school board planned to review the AP program, including the participation rate, in the coming school year.

As Nicholas Colangelo in Iowa and Mike Reno in Michigan can tell you, most high schools in their states are very similar to New Canaan High in the rules that keep students from taking AP. The Newsweek data, and my visits and calls to hundreds of school across the country, suggest this is the case throughout the country. The vast majority of schools don't pay much attention to their AP statistics, and don't think the fact that they limit access to those courses and tests is problem.

But each year there are more school rating scoundrels, toting up the AP and IB numbers and asking why so many students are denied the chance to face a college-level academic challenge. Eventually, I hope, more educators will join the club, and let the numbers help them produce better schools.