Challenge Index

Jay Mathews
Washington Post Education Writer
Monday, May 21, 2007; 12:00 PM

Washington Post education writer Jay Mathews was online Monday, May 21, at noon ET to discuss his Challenge Index rankings of national high schools

A transcript follows.


Jay Mathews: Hi everyone. Why do the annual live chats on the Newsweek list of America's Best High Schools always have to happen on perfect May days? But, duty calls. I welcome your questions and will get started right away.


Charlottesville, Va.: I realize that college is important for a majority of high school students. However, vocational education is just as important for both college-bound and trade school-bound students. It seems to me that vocational education is a red-headed stepchild of the education system, and ranking schools primarily on college related attributes makes my point. Would you consider incorporating a ranking system that also evaluates a school's ability to provide excellent vocational education?

Jay Mathews: It is an interesting idea, although I have yet to see any useful way of rating the quality of a school's vocational education program. And I have one more concern: A study last year, out of the Harvard Ed school I think, showed that the reading, writing, math and presentation skills needed for college were exactly the same skills needed to find good jobs in technical fields right after high school. I don't think AP and IB are just for college-bound kids. There are plenty of future plumbers and auto mechanics who like the give-and-take of a good argument on American politics and want to be able to write with flair and persuasiveness.


Chevy Chase, D.C.: Mr. Matthews:

Please remind your readers why your Challenge Index measures the number of AP/IB tests taken, without regard to the scores received on those tests. After all, who cares if every student takes an AP test if they all receive low marks? Also, please address the disconnect between the high Challenge Index score for Woodrow Wilson High School in D.C. and the low acceptance of that school by in-boundary families. It's my understanding that the majority of students at that school come from out-of-boundary households.


Jay Mathews: I will be glad to. It is an essential point. We do indeed measure the success of each school in achieving passing scores with the Equity and Excellence percent, the portion of all graduating seniors that had at least a 3 on an AP exam or a 4 on an IB exam. But the main list ranks only on the basis of AP and IB test participation, and not scores, because 20 years of reporting, including conversations and observations of hundreds of AP and IB teachers, has convinced me that even if a student struggles in AP or IB and flunks the exam, he or she has gained an experience that will help survival in college. They have gone one on one against the academic equivalent of Steve Nash, and Nash has beaten them, but they now have a visceral appreciation of what it is going to take to compete at that level. That is invaluable, and it also discourages high schools from doing what most of them do---only let their strongest students take AP.

As for Wilson, I am afraid the core reason is racism and classism. Many middle class white people are uncomfortable sending their children to schools with lots of low income minority children, even when the school's program is as good as Wilson's. I have several middle class white friends at the Post who have chosen to send their children to Wilson and are very glad they did. The list was designed in part to help such people look beyond the demographics to the actually quality of the challenge and the teaching at each school.


Arlington, Va.: Ummm, your "winner" this year is clearly playing the game. I'm not sure how any school has an index of more than 14. Assuming seven class periods that means up to seven AP exams (and even that would be unlikely.)

Jay Mathews: That is a small magnet school (actually the top two schools on the list are in the same building in Dallas, and are very similar) that goes out of its way to recruit low-income and minority students in the city and give them one of the strongest doses of challenging courses available in American public education. They start taking AP courses and tests in freshman year, and and since we count the AP tests taken by every student, not just seniors, they build up big numbers. You should visit those schools. They are quite something, and were following this system long before the Newsweek list came into being.


South St. Paul, Minn.: Jay: We often hear that many students, if not a majority than a significant minority are in school just putting in time. Sometimes these students are disruptive to the learning environment. When it is suggested that they be removed from the classrooms the response comes, "Well at least they are safe and off the streets." However, safe and off the streets is different from being educated. I believe this is a national problem that requires a national discussion. What can the average citizen do to help start this discussion?

Jay Mathews: That is a terribly important issue, and goes right to the heart of the debate over reducing the dropout rate in our inner city schools. The solution up to now has been to create separate high schools for students who are disruptive. It works in some cases, if you find teachers who have a knack for connecting with such kids. I also think making all high schools smaller will increase the chances of having at least one adult in each school who develops a good relationship with each kid, and finds ways to engage the ones least comfortable in an academic environment. But I also think that in some cases, letting the student drop out and get a taste of the real world may be the best approach. Many such students come back, or get GED's and set the world on fire, like a recent governor of Delaware.


Washington, D.C.: Why a national ranking? What is the advantage over local/regional rankings - this seems like it would provide an easier way to utilize the list for the purpose of choosing areas to live in.

Also - do you think there is bias towards DC area schools because you are a Washington based reporter - thus making DC/MD/VA schools look stronger compared to areas who may not have as vested an interest in Washington Post reporters?

Jay Mathews: We rank nationally Dow Jones companies, football teams, colleges, hospitals, ice cream chains. Why not high schools? My principal reason for ranking was simply to draw some attention to this important issue---the lack of challenge in most of our high school. (Note that only 5 percent of public schools reach the very modest benchmark to make this list. All a school has to do is have half of its students take one AP exam junior year and one AP exam senior year, and they are on the list.) We primates are tribal, we love pecking orders, and we cannot resist looking at a ranked list. If I didn't rank, no one would pay any attention. Also I thought it was important to signal parents in states that have very few schools on the list that there were these terrific national programs, AP, IB and Cambridge, available to them too.

I rate every school the same way, with the same set of numbers, and so DC area schools do not have an advantage. Although I grant you the school leaders here have more of a chance to read my regular rants on the idiocy of restricting access to AP and IB./


Stockton, Calif.: Of the top rated schools (say 20 or so) how many do not select students. Highland Park in Dallas does not, for instance, but is an island of extremely high socioeconomic status. In other words, which high-ranking schools are non-selective and average or below-average socioeconomically.

Jay Mathews: About half have some kind of selection system, but often it is not based on grades and test scores. H-B Woodlawn is a straight lottery. You can't get into Preuss unless yr parents are low income and did not graduate from college. For the first few years we left off the list schools that selected most of their students by grades and test scores, but then I looked closely at the SAT numbers. Why should Saratoga High in Silicon Valley, with open enrollment for its very wealthy neighborhood, be on the list with an average verbal and math scores of 1275 when Lowell in San Francisco, a magnet school, is kept off the list even though its SAT is only 1236. Or how about Banneker in DC, which selects students but has an average SAT below 1100? So we changed the system and decided to exclude only those magnets (putting them on our distinguished Public Elites list) with average scores above the highest for regular enrollment schools, 1300 SAT or 27 ACT.


Denver: Jay --

There's much criticism of No Child Left Behind on the grounds that it puts too much emphasis on testing -- and promotes "teaching for the test."

Is this critique fair? And would it also apply to your Challenge Ranking?

Jay Mathews: The Challenge Index rating escapes this critique from anyone who has actually read or taken an AP, IB or Cambridge test because these are tests so good that teachers would kill to test for them. Good scores depend mostly on critical thinking and analytical writing--answering essay questions that are graded by outside experts.

The tests used under NCLB are mostly the standard multiple choice, and are less worthy, but i think they provide useful snapshots of what a kid knows. And the research indicates they provide the same results that far more authentic assessments, like expert review of portfolios, do.


AP tests in freshman year?: If AP and IB courses are supposed to offer college-level work, why do you see it as a good thing for those Dallas magnet schools to be administering them to freshmen? To me, this is a logical extreme of what the index is supposed to encourage. I'm all for raising the ambitions and achievements of underprivileged kids, but they shouldn't be doing college-level work at 14. Most high-income 14-year-olds shouldn't be doing college-level work, either. That's what college is for.

Jay Mathews: You are wrong. But it will be very difficult for me to convince you you are wrong until you visit these schools, talk to the kids, watch them in class and see how they do on the exams. Two things to remember: 1. These are college INTRO courses, and college intro courses are not as demanding as the killer advanced college courses you and I remember, and 2. Our smartest and most ambitious kids these days are smarter and more ambitious at age 14 than we were at that age. (Though i am reading a bio of Alexander Hamilton, and in those days it was very common to see students start college at 13 or 14. The human brain has a lot more capacity than we give it credit for, despite our habitual view that 14 year olds can only do so much.)


Arlington, Va.: If I want my son to get the best public school education and have a great shot of going to a top college which public high schools in Northern Virginia should I consider?

I know you like Wakefield (ours now) because it does a great job giving all kids a chance but for this issue I'm not concerned about other kids -- just my own.


Jay Mathews: Wakefield is perfect, actually, because it solves the problem embedded in your question. That is, getting the best public education is not the same thing as getting into a top college. We have data showing that the more competitive and challenging yr high school is, the LESS likely you are to get into the most selective colleges, but the MORE likely you are to get a great education. Yale and Princeton only take four or five kids a year from a school like Sidwell Friends, for instance, so the many other bright kids there have to be satisfied with equally good, but less famous, Macalester or Occidental or Elon.

Wakefield has the advantage of an unusually challenging program with great teachers, but a less competitive student body, since unlike Sidwell half of its students are low income. And the Ivies love to take kids from diverse public high schools. I advise you congratulate yourself on yr great choice of high school.


San Jose, Calif.: What is the "formula" for creating a great school?

I'd argue that it's a mix of teacher quality, parental involvement, funding, educational variety and specialization, and access to broadening activities. It seems like those things would be most prevalent in private schools. Are private schools uniformly better than public ones?

Jay Mathews: They are not, because all of the qualities you name are far less important that the first one you name, teacher quality, and many of our best teachers prefer public schools because they know they are more needed in them. Private schools look good because they have more middle class parents, and their kids are easier to teach. The best teachers want a challenge. And the data show that the test results from public schools are very close to the results from private schools with the same demographics.


Cincinnati: Can you correlate funding levels with performance? In other words, are the liberals right that more money equals better education?

Jay Mathews: It would be nice if it did, but it doesn't. The DC schools spend much more per child than most suburban districts in the country do, but do not get the same results. Low income kids are harder to teach, and require more attention and more time and more good teaching to get up to suburban levels. But it is being done in some DC schools, like the KIPP schools, for less than what is spent per kid in Arlington or Fairfax. It is not money. It is great teaching, that makes the difference. If we could find a way to attract more great teachers with more money, I would be very much for it. But it would require us to regularly assess teachers results, and is hard to do fairly.


Pomfret, Md.: I know the argument for the challenge index is that just being in an AP class helps. My question is on logistics. Do you think having 30 students in an AP classroom is as effective as having 20? Should there be some sort of cap (even if it is soft) on the number of students in an AP class?

Jay Mathews: No. Read my book Escalante, about the best AP math teacher I even saw. He had up to 50 kids in his AP calculus class. If the teacher quality is the same, then a class of 20 is better than a class of 30. But usually you do not have enough quality teachers to make that work. So the great teacher with 30 kids is going to do much better than the average teacher with 20. Class size only starts to matter when you get below about 17 kids.


Alexandria, Va.: Oh, man, do I ever have issues with your Challenge Index, and do every year. My problem with your rating system is that is still just measures how a school does with the best students there. Say what you want, a school is about more than just the top tier of students. At T.C. Williams, a school with which I know you are familiar, there is an open door policy, which is fantastic. But a crucial strength of T.C. is also how it deals with the students who can't take AP test for whatever reason. ESL students, for example. So, while higher participation in AP is absolutely fantastic, it doesn't tell the whole story, and your negligence in addressing this issue is beyond frustrating. A school can be excellent without massive AP participation. How it helps the students who aren't 4.0 students is important, too.

Jay Mathews: That quality is important, but not yet measurable, and TC Williams is a perfect example of a school that has benefited from the values that are part of the list. It used to be very restrictive in the kinds of kids it let into AP. I regularly chided the school, pointing out that Wakefield, with the same majority minority demographic, had twice as high an AP participation rate. Now TC has changed, opened its AP classes to all, made sure all take the tests, and is in my view a better school.


Cleveland: Congratulations Mr. Mathews on your creating and popularizing the Challenge Index. Indeed, the best public school is the one that brings the most out of every student, economically/academically poor or otherwise. Are you constantly fine-tuning your index formula to truly identify deserving schools?

Jay Mathews: You are very kind. I am continually looking at the numbers, such as the low E and E rates at some schools. And I am open to any and all ideas from readers.


Alexandria, Va.: Thank you for the Index. It's been extremely helpful in bringing rigor to many students who were being left out. Has opposition to the Index from private schools increased or decreased recently? And is there a difference between private Catholic schools and other private schools in their attitudes toward and support of the Index?

Jay Mathews: The resistance is about the same, but you make a good point. The Catholic schools, with more average students, are far more willing to give me their data than the most expensive independent schools.


Bethesda, Md.: What does it mean if a school has an Excellence and Equity index of less than one? E.g., Whitman in my area is listed as having an E&E index of "0.7" Does that mean that only seven tenths of one percent of graduating seniors passed at least on AP or IB test? Seems unlikely.

Jay Mathews: Whoops. We will fix that. It is a mistake. It is supposed to say 70 percent. Thanks for the good catch. Whitman's number is excellent. The national average is 14.8 percent.


Princeton, N.J.: Participation in AP tests is affected by many factors, including (a) the not-insignificant cost of taking the AP test and (b) the fact that the very best colleges do not always provide AP credit for AP scores. How do those two factors bias your Challenge Index?

Jay Mathews: The list I think has encouraged school districts to pay the test fees for students who cannot afford them, or in the case of most No. Va. schools, pay the fees for all kids and require them to take the tests. Those are very good things to happen. The college academic departments have proven to be very thick-headed on the credit issue, with almost no data to justify their decisions, but the admissions offices, a very different part of the university, love AP and IB, and that probably has also led more schools to improve access and expand offerings.


Bethesda, Md.: The Challenge Index is often itself challenged for not factoring in how students at each ranked school actually perform on the various tests, and for not considering other quality dimensions of schools that might be more difficult to quantify and compare.

In fact, the fine print in the introduction to the list emphasizes that, "The rating is not a measurement of the overall quality of the school but illuminates one factor that many educators consider important." Nonetheless, in your article a couple of days ago you characterize it as "-the Challenge Index list of America's best high schools . . ."

Question: Is this, in your view, a list ranking the "best" high schools -- or a list that does not reflect a measure of "overall quality" but rather "illuminates one factor that many educators consider important"?

(Personally, I think the ranking is very fair and useful, but captures a single dimension of quality, not summarizing quality overall.)

Jay Mathews: Best is a very elastic word in our society. My list of best movies may be those that earned the most money. Yours may be those that won the most awards. I continually ask if anyone can think of a better single quantitative measure of high schools, and so far i have gotten no takers.


Golden, Colo.: I have heard school administrators complain about the rankings with statement like, "It is unfair to rank a complex institution with a single number indicating its value or worth." Yet isn't that exactly what those same schools do when they provide a student's GPA and class rank?

Jay Mathews: Great answer! I will have to use that. Also, this is a narrow measure, but that is one of its strengths. It makes it very clear what i am measuring.


Anaheim, Calif.: Jay, two high schools from Dallas ranked first and second on your challenge index. Looking through their past performance, both are in the top spots. Is there anything special or unique about the Dallas school system that other high schools should emulate?

Jay Mathews: See my first answer.


Jay Mathews: Thanks very much. Have to run to a meeting. Most of the remaining questions have answers on the FAQ list on the and Web sites, or feel free to query me personally at Gonna go outside and sniff the spring air.


Editor's Note: moderators retain editorial control over Live Online discussions and choose the most relevant questions for guests and hosts; guests and hosts can decline to answer questions. is not responsible for any content posted by third parties.

© 2007 The Washington Post Company