Why I Changed the Challenge Index

By Jay Mathews
Thursday, December 11, 2008 9:45 AM

The minute I saw that Coolidge High School in the District had given a startling 750 Advanced Placement tests last May, and that only 2 percent of those exams had received passing scores, I knew I was in trouble.

For 10 years I have been ranking high schools based on participation in AP, International Baccalaureate and other college-level exams. I call this the Challenge Index. It is the system used by Newsweek in its annual list of top high schools and by The Washington Post in its annual ratings of all Washington area schools, published today in The Post Extra sections and on washingtonpost.com.

Every year I receive thousands of e-mails about these lists, and my refusal to include test scores in the ranking calculations. Some readers praise me for recognizing schools that work hard to prepare students from poor families for college-level courses and tests, even if their scores aren't good. Others denounce me for giving high ratings to schools full of such students, because many people think low scores should disqualify a school from appearing on anybody's best schools list.

I like the nice messages, but the ones from people trying to straighten me out are more fun to answer. I tell them they are right. Average test scores are one valid measure of a school. But those averages are so influenced by the family backgrounds of the students that if you ranked schools by average parental income instead of test scores you would get pretty much the same result.

The Challenge Index gives low-income neighborhood schools with energetic and insightful teachers and administrators a chance to shine by showing their strenuous efforts to raise the academic level of their students, rather than bask in the glory of having classrooms full of doctors', lawyers' and accountants' kids.

I have a drawer full of e-mails from parents, students, educators and pundits saying I am right about this, but the Coolidge High statistics that appeared on my computer screen the day before Thanksgiving were a severe test of that point of view. Few public high schools have academic reputations worse than Coolidge's. A year ago my colleague Lonnae O'Neal Parker described the school in great detail as a chaotic collection of distracted teachers and inattentive students. Yet, six months after Parker's stories appeared, Newsweek published my 2008 Challenge Index list, on which Coolidge, despite its flaws, had a rating of 1.038. That meant it ranked among the top 5 percent of all U.S. public high schools measured this way.

I did some quick arithmetic when the new Coolidge data came in. I divided the number of tests, 750, by the number of graduating seniors, 137, to get the rating. It had jumped from 1.038 to 5.474. That would make it No. 1 in the Washington area. It would be ahead of perennial No. 1 H-B Woodlawn in Arlington, which this year had 399 AP tests and 76 graduates for a rating of 5.250.

As I said, the Challenge Index measures test participation, not test scores. The idea is to see which schools are getting as many students as possible to try at least one college-level course and test. Coolidge's participation rate was close to Woodlawn's. But their passing rates -- the portion of tests with grades of 3, 4 or 5 on the five-point AP exams -- were in different universes. Coolidge's passing rate was 2 percent. Woodlawn's was 59 percent, almost 30 times greater.

Coolidge Principal L. Nelson Burton, like a growing number of principals in impoverished neighborhoods, has decided that one way to rescue his students from the low standards that Parker described so well is to entice as many of them as possible into AP. Like IB, AP exams are written and graded by outside experts. Neither Burton nor anyone else in his school, or his school district, has the power to dumb down those exams. This is very different from what happens with final exams in regular courses, which are often made easier than they ought to be. Schools cannot afford to give out too many failing grades, no matter how poorly their students are doing. Parents would complain to the principal and the school board. Teachers who try to keep their classes, and their final exams, at a high standard are forced to compromise.

Teachers in AP and IB classes still give their students whatever class grades they like; the college-level test results do not arrive until summer, long after report cards have been handed out. But those AP and IB results are becoming public documents, reported on state government Web sites, so anyone can see (as long as all or nearly all AP and IB students take the exams) how well a school is doing when measured against an incorruptible high standard.

Burton and principals like him know they are going to see low scores in the early years of their AP programs. They accept that because they believe their students, teachers and communities should feel the icy blast of real college standards. Their students, they argue, need to struggle with long reading lists. They need to sweat through three-hour exams full of essay questions. They need to get a sense of what college is like that they cannot realistically get any other way. Those school administrators want their AP students to go one-on-one against the academic equivalent of pro basketball star LeBron James. Their kids will lose that game, they know, but gain a visceral appreciation of how much work they have to do to rise to that level.

I admire Burton and those other principals for exposing so many of their students to AP and IB, and some lesser-known but similar programs such as the Cambridge tests. If there were a less demanding but still well-structured and tamper-proof system like AP, IB or Cambridge that could be used to build their students up without so many failed exams, that might be a viable alternative. But there is no such option. The most frequently mentioned substitutes, such as state tests in Maryland or college admission tests such as the SAT and the ACT, have almost all multiple-choice questions. They provide little incentive for developing the critical thinking and writing skills that AP and IB tests demand, and that college students need.

Some of my e-mail correspondents wonder whether it makes sense to prepare a lot of Coolidge kids for college. Perhaps, they say, we are expecting too much of impoverished children. Do those teenagers really need AP or IB to get into trade schools, or good entry-level jobs? Why not prepare them for those life paths instead? College isn't for everyone.

I have two answers to their good questions. First, experts on earning a living wage in the new information age say the thinking, writing and presentation skills needed for college are also needed for the job market. Second, low-scoring schools could prepare many more students for college if they committed themselves to the task. Many high schools with large numbers of students from low-income families have done so.

I put together these pairs of schools in the Washington area with similar percentages of economically disadvantaged students but very different AP passing rates:

Crossland in Prince George's County (42 percent from low-income families, 3 percent AP passing rate); Wakefield in Arlington County (43 percent low income, 53 percent AP passing rate)

McKinley Tech in D.C. (53 percent low income, 5 percent passing); T.C. Williams in Alexandria (50 percent low income, 52 percent passing).

Forestville in Prince George's (47 percent low income, 7 percent passing); Wheaton in Montgomery County, (48 percent low income, 32 percent passing).

Administrators and teachers at high-poverty schools with good passing rates say their success is the result of hard work by teachers, both in high school and in lower grades. They make sure impoverished students without college role models at home receive extra time and encouragement to learn. It takes a few years to establish such programs and bring the standards up in all classes, but it can be done.

Thinking about this, I realized that Coolidge was at only the beginning of its effort to strengthen its teaching in that way. It was going to take the school some time to get to the place where significant numbers of students would be passing the exams. If I left the school high on the same list with schools that had already gone through the process, it would create confusion about what the list was measuring. It would also leave the false impression that Coolidge officials were trying to involve students in AP as a gimmick to look good in The Post and Newsweek, when in fact they were trying to construct the foundation of a challenging academic culture.

Some readers have argued that the Challenge Index is an invitation to unscrupulous principals to stuff students into AP and create a false aura of academic success, but I don't think that is possible. Teachers, parents and students have to accept the AP courses as valid, as many of them have at Coolidge. A principal who tried such a scheme without community support would soon be looking for another job.

In April I wrote a story for The Post about Bell Multicultural, another D.C. high school that, like Coolidge, is bringing AP to nearly every student. Bell's program is remarkable because its two required AP courses are English Literature and English Language, in a school full of immigrants' children where English was not the first language spoken at home.

In 2007, only 13 percent of 311 AP tests at Bell received passing scores. This year, with the number of AP tests increased to 393, the passing rate at Bell fell to 8 percent, as often happens when more students participate. But support among parents and students was nearly universal. Manuel Ventura, a senior who did not get a passing score on his AP English Literature test, told me: "I really think it is a great opportunity for people like me. I feel proud of myself, and I thank all my teachers."

The day after my article, Rachael Brown, a recent teacher at Bell, posted a long comment on the DCist blog. She said I did not emphasize enough the difficulties of getting Bell students to a level where they could handle AP courses and exams. I kept reading, wondering if she would suggest the school abandon the program.

She did not. Keep the AP requirements, she said, but work harder to get everyone ready. "There have to be supports, lots of them, in place for struggling students, and safeguards to make sure the highest-performing kids aren't being slowed up," she wrote. "Transition takes time; it's messy and makes more work for everyone, but is worth it in the end. As one student told me, 'I never thought I could learn this stuff.' That same student has already been accepted to three colleges."

Bell is raising students' expectations for themselves, as are Coolidge, Friendship Collegiate, Thurgood Marshall, McKinley Tech, Cardozo and other D.C. schools willing to accept low scores at the beginning to build their AP and IB programs. Each year a few more schools adopt this approach, creating a new species of high school college-prep, using the most demanding high school courses in the country to help them catch up with schools that have had AP and IB for years.

Because schools such as Coolidge are so different from other AP and IB schools, I decided to create a separate list for them. I am calling it the Catching Up list. It includes all schools with AP or IB test passing rates below 10 percent. Some schools on the new list have moribund college-level programs and, sadly, don't seem to be trying to catch up with anybody. But they are at the bottom, where they should be. Schools with ambitious programs, such as Coolidge, are at the top.

This is the first time in the history of the Challenge Index in which we have had significant numbers of high schools with very high AP participation rates but very low AP test passing rates. There were none in the first lists that appeared in the Post and Newsweek in 1998. A Catching Up list then would not have been possible.

This will, I imagine, spark much discussion. My e-mail queue will be full. Many readers will be critical. Some students, parents and teachers at the schools on the new list might accuse me of slighting their efforts and creating a caste system. One erudite principal said the Catching Up list reminded him of Plessy v. Ferguson, the 1896 U.S. Supreme Court decision that endorsed allegedly separate-but-equal school segregation.

Readers who have long advised me to remove low-scoring schools from the list altogether might interpret this change as acceptance of their point of view, and ask why I lack the courage to go all the way. This thinking is part of the new U.S. News & World Report "America's Best High Schools" list, inspired by Education Sector think tank co-founder Andrew Rotherham, with whom I have discussed the issue many times. Some might suggest that I am tilting in Rotherham's direction, but they would be wrong. He thinks that schools like Coolidge, Bell and Crossland should never appear on any list of the nation's best schools, while I think they deserve public recognition, which the Catching Up list provides, for their strenuous efforts to change their cultures.

The Challenge Index grew from my anger at a serious shortcoming of high schools that few people -- and no journalists -- were paying attention to. The problem is that most average and below average students are barred from taking AP courses and tests, even though the experience of many AP teachers, and some research, indicates those challenging experiences would help them prepare for college and for jobs.

The Catching Up list is one more in a series of adjustments to the list. From the beginning I excluded schools such as the Thomas Jefferson High School for Science and Technology in Virginia and Whitney High School in California that were so selective that they had few or no average students. They did not belong on a list designed to show which schools were challenging average and below average students. Newsweek has created a separate Public Elites list for them. I exclude private schools because so few of them are willing to release their AP and IB data. New schools are also kept off the list until their senior classes reach full size so as not to inflate their ratio of tests to seniors.

Schools on the Catching Up list that reach 10 percent passing rates will be moved to the main list. The same rule applies to IB, but so far there are no Washington area schools that use IB as their principal college level test that have passing scores so low. I will suggest to the editors of Newsweek that they create a separate Catching Up category when they next publish their national top high school list.

A 10 percent passing rate -- the dividing line between the main list and the new list -- seems very low. But in a school giving many tests, that translates into a significant number of AP students earning college credit. Few students at most high schools take more than two tests. If 10 percent of 750 tests at Coolidge had received passing scores, that would have meant 35 or 40 students qualified for college credit, about the average number for AP schools nationally.

Creating two lists is not an exercise in segregation. It is a way to put more emphasis on passing the tests and creating more incentives to do so. Professional soccer teams in Europe move from lower to higher leagues when they complete successful seasons. The same rules apply here.

Educators and parents at some Catching Up schools said they had no problem with the change, or what their schools were doing. Terry Goings, a parent leader at Coolidge, said: "I am more for preparing people for college than passing a test, and that's why I support AP courses." Arsallah Shairzay, dean of early college and Advanced Placement programs at D.C.'s Friendship Collegiate, said: "The student needs to be given a challenging environment." Alexandra Pardo, academic director at D.C.'s Thurgood Marshall, said: "If a student wants to take a rigorous course, why deny him or her that chance?"

This is a complicated issue that can be best understood through in-depth discussion. I will be doing a live chat on washingtonpost.com at noon Friday to field questions from readers. Those of you who think I have made a mistake should log on and come after me, and see if I can persuade you otherwise.

Editor's note: Class Struggle usually appears Fridays on washingtonpost.com. We are publishing this week's column a day early to coincide with publication of the new Challenge Index rankings of local high schools in The Post's Extra sections. Next week the column will resume its regular Friday morning schedule.

View all comments that have been posted about this article.

© 2008 The Washington Post Company