Wednesday, Dec. 13, 2006 at Noon ET

Challenge Index

Andrew J. Rotherham and Sara Mead
Education Policy Analysts
Wednesday, December 13, 2006; 12:00 PM

Education analysts Andrew Rotherham and Sara Mead"believe that the Challenge Index is a seriously flawed measure of overall quality." They'll be online Wednesday, Dec. 13, at noon ET to discuss their objections to The Post's high school rankings and offer their thoughts on how to measure school success.

Submit your questions and comments before or during the discussion.

Rotherham is co-founder and co-director of and Mead is senior policy analyst for Education Sector, a nonprofit education policy think tank. Rotherham was a special assistant to President Clinton, advising him on education policy, and he serves on the Virginia State Board of Education. Mead worked for the Dept. of Education and the Gore2000 campaign. She lives in the District and has studied District schooling policies.


Andrew J. Rotherham: Hi,

Thanks for having us. For background for readers, the original analysis of the Challenge Index that Sara and I did can be found at the link above.


Skewed Index: I am all for college prep, AP and even IB programs. However, my nephew is in a college prep program and takes two AP classes. I tutored him in algebra and cannot fathom how he is ready for AP physics and calculus. His parents say he is struggling in the classes as well. He is on pace to graduate, as he has his base credits covered. However, for the index, he counts as a positive. Wouldn't the index be better if it measured the ratio of number in classes and students, and weighted it by the percentage that passed the exam? There are other ratios to examine as well. This index is really just a participation index. This could merely reflect which districts have made the resources available versus which students benefit most from the programs and tests.

Sara Mead: You are right that AP test-taking doesn't measure whether or not kids actually mastered the AP curriculum. One of Jay's goals with the challenge index has been to encourage schools to open AP opportunities to all students who might benefit from them, rather than restricting them to just students that they judge are most likely to pass an AP test. There's some good logic here: evidence shows that kids benefit from being exposed to the rigorous academic curriculum that well-taught AP classes offer.

But you are also correct that this means a school can do well in the challenge index even if its actual AP classes are not of high-quality and they kids don't do well on the AP tests they take. That's why the College Board developed a measure, which Jay added to the challenge index this year, that measures the percentage of all a school's graduating seniors who had a passing score on at least one AP exam. It's not a perfect measure (I'd prefer two separate indicators on what percentage of a school's students took at least one AP test and the percentage of students taking tests who passed them), but it does improve the information that's available about how well a school is doing preparing its students to pass the AP exams. You can find those ratings for D.C.-area schools in the Post package on the Challenge Index.

One minor detail worth noting here: The challenge index is based on the number of kids taking AP tests, not the number enrolled in AP courses. A lot of kids take AP courses but don't take the exam, and some kids take AP exams in subjects for which they have not taken an AP course (often because their school doesn't offer such a course).


Springfield, Va.: If the Challenge Index is inappropriate, what do you recommend parents look to when trying to objectively select a school district? At this time I am contemplating relocating my family from Springfield, Va., in the Lake Braddock school district, to Stafford, Va., in the Colonial Forge district. Which would you choose, and why? Thanks.

Andrew J. Rotherham: Hi,

As we said to the previous question on this, check a variety of sources and then make the best decision you can. Bryan and Emily's book will help you with that if you want a resource. But don't just trust the realtors, there are a lot of factors that matter in terms of "fit" for your child and you should weigh them all.


Fairfax, Va.: Doesn't the size of the school work against it in the Challenge Index? If the school has a very large senior class it can lower its challenge index score.

Sara Mead: Nope. The Challenge Index is a ratio of the number of kids in a school who take AP tests to the number of a school's graduating seniors. If a school is large, that (hopefully, although not always) means it has a large graduating class. But it also means it has a large number of potential test-takers. So, we should expect a big school to have more kids taking AP or IB tests than a small school. If the index just measured the number of kids in a school taking AP tests, it would work against small schools.

(It's worth noting, though, that the Challenge Index does not measure the percentage of a school's graduates or students who took an AP test.)


Arlington, Va.: Hi Andy,

Katryn Nicolai here. Can you summarize why the Challenge Index is not a reliable tool?


Andrew J. Rotherham: Hi Katryn,

Our concern with the index boils down to what it does not take into account rather than what it does. We agree with Jay that AP course-taking is very important. However because the index doesn't look at other measures, for instance graduation rates, scores on state standardized tests, etc it gives a very incomplete measure of school performance. Consequently, a school can have a few students taking a large number of AP courses while most students are getting a subpar education and it can still rank as one of the "best" schools under this index. In fact, perversely, a high dropout rate can actually help a school's ranking by changing its denominator in Jay's formula. We think that's unacceptable considering the nature of the educational challenges the region and the nation face today.


Washington, D.C.: Your critique seems to rely heavily on the existence of various forms of "achievement gaps" which are certainly an important policy issue. But from the perspective of an individual whose children are on the "good" side of these gaps (most Newsweek and Washington Post readers, for example), what's the problem with using an index that ignores them as an aide for deciding where it might be good to send one's kids?

Andrew J. Rotherham: Hi,

Even in the affluent suburbs that surround this city, there are substantial gaps in achievement for students by race and income so I'm not sure I'd characterize the Post's readership in exactly that way. In fact, on average, African-American students perform better on Virginia's standardized (and horribly named) SOL tests in Richmond than in Fairfax County. You won't see that on the real estate brochures.

So while AP offerings are one thing parents should consider when they think about what high school to send their children, too, there are a host of other factors they should consider to. And Sara and I think that it's incumbent on the Washington Post to do that and incumbent on Jay Mathews to make Newsweek's list of "best" high schools better approximate a reasonable definition of that word.


Ft. Washington, Md.: What advice can you give to parents that live in an area (P.G. County) with the lowest-ranked schools but cannot afford to move or send their children to private schools?

Sara Mead: I am assuming that you are talking about schools that are ranked low on the Post's Challenge Index. I know that Jay Mathews, who developed the Challenge Index, would agree with me that just because a school doesn't do well on the challenge index, that doesn't mean it's a bad school. There are lots of schools that do a good job educating their students but do not necessarily offer AP courses or encourage their students to take AP tests. Conversely, evidence from state assessments, graduation rates, and other measures shows that many schools that did well on the challenge index don't actually do a good job of educating their students, for reasons Andy describes above.

The challenge index is only one, and I would argue one very narrow and flawed, measure of how well schools are educating their students. That's one of the reasons Andy and I have challenged calling schools that do well on the national list America's "100 best high schools."


D.C.: Sara, you've done important work on gender in education, how does that factor into the Challenge Index?

Sara Mead: Thanks for saying nice things about my work! We didn't look specifically at achievement data for boys and girls in schools on the challenge index. I would really like to see Jay include more information with the index about what percentage of students in a school from different subgroups (such as students from different racial and ethnic groups) took AP tests. Obviously, it would be interesting to include information on the percentage of boys and girls who took AP tests in that information. In general, girls are more likely to take AP tests than boys, but there are also significant differences depending on the subject. Girls are more likely to take AP tests in foreign languages, whereas boys are more likely to take AP tests in physics and computer science.


Bethesda, MD: You weigh in on federal and state policy on a regular basis. What should the government be doing in this situation?

Andrew J. Rotherham: Great question.

In too many places it's still easier to find out good comparative information when choosing a washing machine than it is when choosing a school. That's a problem that government can solve by making transparent information available.

Information is, of course, power and for too long state and local officials have been reluctant to part with the power that comes with having all the data. But as these new Web sites show, the genie is out of the bottle on transparency and so state and local officials should get in front of it.

Very much related, too many parents still have trouble visiting schools and watching classes when they're making a decision. Obviously schools need to be able to maintain orderly environments where kids can learn and it can't be a free-for-all. But our public schools are public institutions and they need to live up to that charge by welcoming the public in to see what's happening and get a first-hand sense when they're making choices about schools.


Arlington again: Are you going to, or have you, come up with your own list based on your own criteria to rival Matthews' list?

Andrew J. Rotherham: We are not specifically, but I have been working, along with my colleague Mike Goldstein who founded the Match Public Charter School in Boston on a project with a large news organization to do exactly that. Look for it before too long. Sara can speak for herself but she thinks that this whole list business is trouble so she's put her energies into slaying some other dragons lately.


Bethesda, Md.: How would you reform the Challenge Index to better reflect the quality of high schools?

Andrew J. Rotherham: For starters we would include other measures, achievement gaps (or lack thereof) and graduation rates (again disaggregated by race and income). As Jay points out, when he started the index information like this was not available in a comparative way. That's changed though and the Challenge Index needs to change, too

_______________________ Match School

Andrew J. Rotherham: Here is a link to the Match School. It's a very interesting school, best open-enrollment high school in Boston, and an example of what can happen when all kids get a rigorous curriculum


Hagerstown, Md.: Isn't this ranking unfair to smalls schools in rural areas that can't offer as many AP classes as big city and suburban schools?

Sara Mead: It's probably easier for bigger schools to offer a lot of AP classes, but there are plenty of ways smaller and rural schools can help more kids take rigorous college-level courses, too. One of the easiest ways is by encouraging more students to take college-level courses at their local community college. It can be an affordable way for rural communities to offer their students courses like calculus or physics for which it's often hard to hire qualified high school teachers. Former Virginia Governor Mark Warner supported initiatives to encourage this. Another thing some small schools can do is to make AP courses the default course: For example, you could replace your current senior English course with AP language or AP literature, but you would need to do a lot of teacher professional development and improve the preparation kids are getting in earlier English classes.

Students can also take AP tests even if they don't take an official AP course. I was lucky when I went to high school to have several teachers who were willing to work outside of class with me and a small group of students to prepare us for the AP tests, even though our school didn't offer an official AP course. There are lots of ways to expand opportunities for all students to take rigorous college-level work if they are creative.

_______________________ Picky Parents

Andrew J. Rotherham: Due to a technical snafu an earlier question got deleted, it was from a parent wanting to know how they should choose a school if not using the Challenge Index.

In short the answer was, comparative data about schools as well as qualitative measures about the school, informed opinions of others, needs of your child, and careful consideration about what school will be the best fit. And trust your gut. Worry less about labels like public, private, and charter and more about what school is the best fit for your child.

Bryan and Emily Hassel have written a book, The Picky Parents Guide, referenced above, that is a great resource for parents. And Web sites like and offer data for parents and offers parent reviews, too.


Andrew J. Rotherham: Here is a link to the site.


Andrew J. Rotherham: And here is Schoolmatters. It's a project of S& P, and a sign of the times, the same analytic rigor they bring to rating companies and states, they're bringing to the education space.


Alexandria, Va.: Do you think that the emphasis on taking AP classes is putting the squeeze on those students who may be above-average but aren't quite up to the work required in a well-taught AP class? For example, my son's high school is eliminating most "honors" classes in the upper grades, which means you either have to take the "regular" class (often with unmotivated students and a not very challenging curriculum) or take the AP class. There is no middle ground. Without the honors classes, my son has a choice between classes that are too easy (and probably boring) and classes that he may not be able to handle.

Sara Mead: I think some proponents of AP class would probably argue that the "squeeze" you're mentioning is a good thing, because it encourages schools to raise standards and expectations for their advanced students who were previously in honors classes. I have a lot of concerns about the challenge index, particularly because some schools that do well on it don't seem to serve disadvantaged students well.

But if you look at some of the schools Jay has written about, like YES! College Prep, that do serve a lot of disadvantaged students and get them to pass AP tests, it's clear that a lot of kids who a lot of people might think aren't quite up to the work actually can succeed in AP courses with the right instruction and support.

AP courses are rigorous, but they are intended to cover the kind of content in the kind of depth and speed that kids will have to encounter in their first year of college. I'm not sure we do any favors to students who are college bound by putting them in courses we call honors (which assumes they are challenging, college-preparatory work) but that don't prepare them for the rigors of college-level work. Some of these students will get a rude awakening in college. We should also improve expectations in a lot of "average" courses so that they are challenging for students.


Sara Mead: Thanks for the questions everyone. It's been great chatting with you, but now it's time for us to eat lunch. Please check out our work at


Editor's Note: moderators retain editorial control over Live Online discussions and choose the most relevant questions for guests and hosts; guests and hosts can decline to answer questions. is not responsible for any content posted by third parties.

© 2006 The Washington Post Company