A Teacher Takes Me On

By Jay Mathews
Washington Post Staff Writer
Tuesday, July 25, 2006; 11:52 AM

Mark Crockett, a very savvy and energetic social studies teacher at Western Albemarle High School near Charlottesville, Va., has been among my favorite, and most critical, e-mail correspondents for some time. After our latest exchange of messages over my way of rating schools by AP and IB participation, used by Newsweek in its America's Best High School list, I asked if he would be willing to have such a discussion for this column.

He said yes. The result is below. I think it is a very helpful to see how a working educator reacts to what are in many cases very complex issues. You also get a bleachers-eye view of me struggling to field Mark's line drives, bunts and high hoppers. I always learn a great deal from good arguments, and I think this qualifies as one:

Mark: I've been reading your Washington Post columns and articles, and your Newsweek articles, for a good many years now. In fact, I've even used a 2001 article you wrote on racial bias in special education classes as part of a psychology test. But I think your articles on Advanced Placement (AP) courses and tests dramatically overstate their effectiveness and worth.

Jay: In what way?

Mark: In what way? From reading your stories on AP over time, it seems to me that you oversell AP courses and the benefits of AP, and you underreport what research indicates about AP. Thus, the public gets a skewed picture of what AP is and, perhaps more importantly, what AP is not.

Jay: I am guilty of many sins of omission as a reporter, but holding back AP research information is not one of them. If you can cite a reporter who has come even close to writing as much about AP research as I have, I will give $50 to your favorite charity. I was the first reporter in the country to report the Geiser UC study that showed for California students, just taking an AP test, but not passing it, did not correlate with college success. I was the first newspaper reporter and online columnist to report that the National Center on Educational Accountability (NCEA) found the same results in Texas, although did not rule out a beneficial effect from just taking the test. I was the only reporter who gave space to AP critic William Casement's scholarly article arguing that AP courses are not as good as college introductory courses, I let Casement be a guest columnist to make his case in his own words. I have repeated those findings several times in The Post, on washingtonpost.com and in Newsweek.

I realize more than anyone that AP does not get as much attention in the press as it deserves, but I don't think it is fair to say I don't give a complete picture. I do think AP and IB are the best programs we have in U.S. high schools, but I have always told the whole story. What have I left out?

Mark: Before answering the question "What have I left out?" I want to address your statement that AP doesn't get the press it deserves. You have written continually about AP in The Washington Post and online for years. Newsweek devotes a cover story to it each year. That publicity has led to an explosion in AP course-taking with 1.3 million or more students now taking AP courses. And, colleges now give special weight to AP course-taking. AP has become the course de jour in public high schools with school boards and principals rushing to add more AP courses, and you say it doesn't get enough publicity??

What have you left out -- or perhaps embellished? In your recent Newsweek article ("Why AP Matters," 5-8-06) and in some other articles you referenced Saul Geiser's study in California. In citing that study you say good AP test scores increase the chance of earning a college degree but you omit that the study found the best predictor of college performance to be unweighted high school grade point average (UHSGPA) . . . and as you know, most high schools give either a half-point or a full point bonus for taking AP (that's part of the reason kids take AP . . . to help enlarge their GPAs). In discussing the Klopfenstein study (2005) in Texas [a separate study from the NCEA study], you've omitted that study's criticism of AP-sponsored research (" . . . close inspection of the studies cited reveals that the existing empirical evidence regarding the benefits of AP experience is questionable.").

Perhaps your most egregious omission was in the Feb. 17, 2006, article in The Post reporting on Clifford Adelman's "The Tool Box Revisited." You've cited Adelman's original "Tool Box" for years (wrongly I might add) as proof that AP works. "In Tool Box Revisited" Adelman not only did not find any AP benefit, but also he scolded those who misrepresented his original "Tool Box" study (" . . . a spate of recent reports and commentaries on the Advanced Placement program claim that the original "Tool Box" demonstrated the unique power of AP course work in explaining bachelor's degree completion. To put it gently, this is a misreading."). But your Post article said not a single word about any of that.

Lengthy, I know . . . but you asked. Did I earn the $50 for charity?

Jay: Sadly, no, because the wager was to find a reporter who has come even close to writing as much about AP research as I have, and all the pieces you cited were written by lonely little me.

That is a problem. Some reporters have looked at this subject occasionally. There were several pieces on the occasion of AP's 50th anniversary. But Mathews the-AP-and-IB nut is usually left alone to do most of the hard labor.

I grant you I don't mention every detail of every research report in every piece I write about AP. If I did, it would drive readers crazy. But all those facts I cited that can be used to contradict my support for AP were first reported by me and have been repeatedly reported by me, if not in every single story or column. For instance, in the latest Newsweek piece I once again quoted several critics of the Newsweek list, and the focus on AP and IB, and treated them with respect. That is my duty as a reporter.

As for your other good points, I am in regular communication with Cliff Adelman, I show him everything I write about AP that mentions him, and he has never asked me to change a thing about the way I refer to AP in his studies. I say that his studies are not about AP, but do include AP among the package of intense academic experiences in high school that correlate with higher college graduation rates. I was a bit sloppy in my wording on this question three or four years ago, but Cliff never said anything about it and I have tightened up since.

I have reported the Geiser findings, but noted that they only apply to the top 12 percent of California high school students, not the group that I am most interested in. It is the B and C students and the effect of AP on them that interests me. And I regularly note, because I think it is important, that both the Geiser and Texas studies show a positive correlation between passing an AP test and college graduation rates. I have reported the Klopfenstein study in detail, noting the parts that contradict my view that AP is important, but have also noted -- which you fail to mention here -- the fact that she was looking not at AP tests, but at participation in AP courses. That is a risky way to assess AP, because if the kid does not take the test, he is unlikely to have gotten the full AP experience. The Geiser and NCEA studies that look at AP test-taking are much better measures of what we are trying to spotlight in the Newsweek list, which measures test-taking, not course-taking.

Whew. What's next?

Mark: Okay, you say you've been in touch with Adelman ("Tool Box," and "Tool Box Revisited") and he's never said anything about what you've written. Others certainly have, though. I have. And that doesn't excuse your glaring omission that the "Tool Box Revisited" found no effect for AP. It seems to me that if you've cited Adelman again and again as supporting the value of AP courses and tests -- even when you add in that AP is only part of a rigorous high school curriculum -- then you have an obligation to say that his most recent report didn't find that level of support for AP. You said not one word.

Jay: Adelman did not say AP had NO effect. He said it has less effect than other measures. The Geiser and NCEA studies show that AP had significant effect, if students do well on the tests, and those are studies that focus directly on AP and have much larger samples.

Mark: Yes, Jay, Adelman does say that "AP is only a component of the academic curriculum intensity in both the original 'Tool Box' and the replication" ["The Tool Box Revisited"] (p.40).

But he also says that "AP was tried out as one of the three variables that might serve as proxies for curriculum intensity." Adelman tested statistically for AP, for a math-science combination, and for foreign language study and concluded that "[n]either Advanced Placement nor foreign language study reached the threshold level of significance. . . ."

As for Klopfenstein and Thomas, yes, they only looked at AP courses, and not AP scores. They explain on pages 8 and 9 of the study why that is, and I think I've already referenced that. Geiser and Santelices were looking at U-Cal admissions, a rather prestigious public university, hence the "top 12 percent" figure you cite. But Geiser and Santelices also pointed out (1) AP courses are weak "predictors of college outcomes" (I know . . . I'm citing courses, not scores . . . do you think the general public differentiates these?) and (2) AP courses are the most widely used 'honors' courses in California, and (3) increasingly, AP courses -- not tests, since test scores are not available at the time of admittance -- are the currency of choice for admission to selective colleges and universities. Isn't this really why many students take them? And what of the Geiser finding that unweighted high school grade point average is the best predictor of college performance??

Jay: I am all for using unweighted GPA in college admissions, but the Challenge Index is not focused on college admissions (although it is relevant to it) but college completion. You are absolutely right that the general public does not differentiate between AP courses and AP scores. That is an important reason for producing the index, so they will begin to see the importance of that distinction.

One of the most interesting things I discovered when I first began experimenting with this way of measuring high schools that was some very well regarded schools in affluent neighborhoods, such as Darien High in Connecticut and Shawnee Mission East High in Kansas, had surprisingly low ratings on the index. I called those two schools and discovered what I called the sham AP school phenomenon. They had plenty of AP courses but only encouraged the best students in them to take the AP tests, so there was no incentive for most of the students in the course to learn at an AP level, and likewise not much of an incentive for the teachers to teach at an AP level.

This turns out to be a problem nationally with AP. An AP course in which many students don't take the test appears to be much less challenging than an AP course in which most or all of the students take the exam. That is why the index counts tests, not students in the course. And that is why Klopfenstein's and Thomas's analysis doesn't work well for me, because they are overlooking that distinction. Adelman also is looking at just course participation, not tests. The Geiser and NCEA studies suggest that if he looked at AP tests with scores of 3 or above he would see AP rise to the level of significance in his calculations.

Mark: Since you mention the National Center for Educational Accountability (NCEA) citation, I have several comments about that. First, I've only read a partial selection of their data, so I'm treading on somewhat thin ice here (I've asked them several times for the full report and had no response . . . maybe you can help here). Second, they avoid showing in the data I have read that the overall AP test scores have declined as the number of AP tests has increased (more about this later). Third, I think their data is suspect in that they promote payment to students for taking AP courses and tests, and they support payment to teachers (and administrators?) for helping students attain a "passing" score on AP tests. What we know about the motivation to learn suggests this idea is -- longer-term -- counterproductive. Finally, and I'm going out on a long limb here, I give serious pause to data collected and disseminated by a group started by the former Texas Republican Party Chair (for nearly a decade), who was and remains an avid supporter of George W. Bush (in his campaigns for Texas governor and president), and who has unabashedly favored and advocated the No Child Left Behind legislation.

Personally I do not think the answers to public education's problems is to test, test, test . . . AP or otherwise.

Jay: What makes you think there is a fuller report? The one they issued seemed as complete as the other ones we have discussed here.

Overall averages on any standardized test taken by choice almost always decline when you get more test-takers, since the newbies are usually from lower-income groups and are often less well prepared. Tell me more about what you see is the importance of this in our debate. The AP average has not declined nearly as much as some critics expected, leading them to suggest the College Board is cooking the books or dumbing down the tests, but there is no evidence of that. Good teachers are doing good work. The tests are written by college professors and such a massive conspiracy in higher education to dumb down AP would have been exposed long ago, if it existed.

As for doubts about any research out of Texas, I bet we could figure out a connection between Profs. Klopfenstein and Thomas and some unsavory Texas politicians if we worked hard to do that. Texans appear to be very neighborly and seem to talk to each other all the time, even across party lines. You are more likely to preserve your sanity and win this argument if you just focus on the results of their work, and not their connections. Robert Oppenheimer had a few communist friends, but the important thing was that his bomb worked.

Mark: Yep, the bomb worked . . . and my point about the political connections in Texas, Jay, is simple: does public education work? The best research on that question that I've come across comes from Gerald Bracey and David Berliner et al, and is perhaps best summarized in what is called The Sandia Report ("Perspectives on Education in America," Journal of Educational Research, April 1992). Coming on the heels of A Nation at Risk (which was far more politics than it was education) researchers at Sandia National Laboratories investigated the status of public education in the United States and found "steady or slightly improving trends" on nearly every measure (I'd encourage readers of this discussion to obtain a copy of that report, which the first Bush administration tried to suppress . . . and also to investigate the Eight-Year Study, another landmark report that few educators and citizens seem to know about). So while the conventional wisdom seems to be that public education in America needs fixing, the reliable research says it works pretty doggone well. That's not to say that there aren't problems. There are, especially for low-income and minority students. But the "fix" is not just to throw AP tests into schools and at students. The "fix" requires systemic attention, and it isn't cheap. Luce and O'Donnell (NCEA in Texas) and Karl Rove (Republican political strategist and dirty tricks specialist) have long and deep connections . . . and quite honestly, I don't believe public education is a priority of theirs.

Jay: I agree entirely with the Bracey, Berliner and Sandia assessment of American education. We are doing pretty well by 70 percent of our students, although we could do better with them, and one place where the improvement would be fairly easy is opening up access to AP and IB. It does not take much money, and it can make high school significantly more challenging and thus more interesting for many students, as well as help solve our college drop-out program.

Our real problem is that bottom 30 percent, and I think AP and IB can help there also by telling kids, parents and teachers in inner city schools that they, too, can aspire to the level of learning found in suburban schools. But it is MUCH harder to get low-income kids to that level, and the work should begin in the elementary and middle schools, which is why I am spending so much time looking at the KIPP schools, apparently the best thing going for inner city middle schoolers.

Mark: Jay, okay, so we agree on the Bracey, Berliner and the Sandia research (although we seem to differ on what Geiser, Klopfenstein and others say about AP effectiveness). This is important, since Sandia, for example, shows the U.S. awards more bachelor degrees than any other country, that spending on regular public education has remained constant for decades, that teachers in the United States are paid less than in many other developed nations and teacher salaries in the United States, in constant dollars, are basically the same as they were 30 years ago. And we agree that there are pockets of problems in public schools, mostly with low-income and minority students.

Here is what I see as a significant problem with AP: too many people perceive it as an inexpensive panacea to public education's problems (perhaps this is why conservatives latch onto AP as "the answer"). Is there a problem with motivating students? Bam! (Sorry, Emeril.) Advanced Placement. Achievement gap? Bam! AP. Want to encourage low-income and minority students to attend college? Bam. AP.

I don't think we can correct our problems in public education so simply. H.L Mencken once said that there is a simple solution to every complex problem, and it is usually wrong and makes the problem worse.

Perhaps, Jay, we might also agree that public education in the United States should return to and emphasize one of its core functions: democratic citizenship.

Jay: Compared to a lot of other things we need to do, such as improving teaching in the inner city, opening up access to AP for all motivated students IS pretty simple. You just announce that henceforth, the only requirement for an AP class is a willingness to work hard. No minimum GPA, no teacher recommendation. Motivation rules, and if you don't work hard, you are dumped back into the regular class.

You will have to have more AP teachers if you do this, but I have found this is not such a big problem either. Plenty of teachers are ready for that challenge. And both political parties are trying to promote more AP. In the last presidential campaign, Democratic candidate John Edwards, who is leading the 2008 polls in Iowa, had a very strong pro-AP position. In the 1990s, no two political figures did more to increase funding for AP than Democratic Sen. Jeff Bingaman (N.M.) and U.S. Secretary of Education (and former Democratic governor of South Carolina) Richard W. Riley.

I think you are wrong to say that "too many people perceive it [AP] as an inexpensive panacea to public education's problems." It is not a panacea, just a significant improvement. It will make a difference in many students' lives and probably help our college graduation rate. (We do award more bachelor's degrees than any other country, but several countries have caught up with us in the percentage of young adults holding bachelor's degrees.)

The problem is that the vast majority of American educators not only do not see AP as a panacea, they don't even see it as a useful part of a high school education for most students. About 40 percent of U.S. high schools have no AP courses, and more than half of students going to college have never taken an AP course. It is way off the radar screen, and that's not good.

Mark: Look, Jay, just because some politicians have climbed aboard the AP bandwagon doesn't mean that AP is what you claim it to be. Politicians too often opt for the quick, "easy," and poll-popular approach to solving problems, and I think that's especially true in education. Politicians, at one time, embraced eugenics. I continue to hear politicians cite SAT scores as evidence of successful or inadequate teaching, and the SAT measures virtually nothing.

No Jay, too many educators (and parents and others) DO see AP as a "fix-all." What else explains the huge explosion in AP course (and to a lesser degree) and AP test-taking? And, as I've pointed out before, your Newsweek "Best High Schools in America" ratings help to fuel that growth. The thinking is AP is better (my contention is that it isn't). And it is fairly easy and inexpensive to implement (perhaps this is why conservatives love it) without expending time, energy and money on alleviating school crowding, correcting public school funding imbalances between affluent and needy school districts, or attacking poverty and illiteracy.

You say that AP is a "significant improvement" in public education. But the proof is not in the pudding, as it were. I've already cited the core findings of the NRC (2002) study, the Geiser (2004) study, and the Klopfenstein (2005) study that fail to support that statement. Moreover, Sadler and Tai (2006) study, co-sponsored by the U.S. Department of Education, the National Institutes of Health and the National Science Foundation, "found little evidence that high school Advanced Placement (AP) courses significantly boost college performance in the sciences." AP test scores seem to convey little advantage in college study either, at least in terms of grades (the differences between those who score a 4 or a 5 on an AP test and those who score a 3 or took a non-AP science course are small), since Sadler and Tai conclude that half the variation in scores is due to "background variables unrelated to taking an AP course."

Finally, I'll conclude with a segment from a letter to Newsweek (5-22-06) written by a high school student responding to your AP Challenge Index ratings:

" . . . why am I taking six of them (AP courses) in my senior year? Simply said, I have no other choice. My public, non-selective high school, falling victim to the AP mania that such rankings feed, has replaced most of its honors classes -- where stimulating discussion and debate had reigned -- in FAVOR of AP courses, which are never-ending nightmares of test preparation."

I'd say that sums it up pretty well.

Jay: The greatest growth in AP in the last several years has been in suburban schools, and the motive there is clear. It isn't because all those students, parents and educators think AP will fix American schools. It is because they know the selective colleges now require that their applicants take some AP or IB courses, or explain why not. That is why the student you quote is feeling under such stress, but I have watched a lot of limp courses labeled as 'honors' courses during the past two decades, and I wager that a scientific sampling of high school students would not accept his view that they are better than AP.

There is also more growth of AP in urban and rural schools, because good educators feel their kids deserve a challenge. We even had the ACLU sue the state of California to get more AP into low-income schools, and that seems to me to be a good thing. All those other problems you mention are critical, and must be addressed, but why not add more AP while we are at it? If your child is sick and has to go to the hospital, you take care of that, since his health is the most important thing, but you don't stop reading to the kid at night just because the reading is less important than helping the kid get well.

We have already discussed our very different interpretations of the NRC, Geiser and Klopfenstein studies, and don't forget the NCEA study. But I think you should wait a bit before citing the Sadler and Tai study, since it does not yet exist in written form. I have asked for a copy and expect to get one once it is published, but at the moment it is just a few powerpoint slides and a Harvard press release. There are some issues the study appears to address that are pretty interesting, but to judge from the press release, its conclusions on AP don't add much to the discussion. They are based on a sample of only about 500 students, and show different results than the Geiser and NCEA students, whose samples are 100 times as large.

Thanks very much for your continued attention to these important issues, and for your excellent points and questions. I will try to be more careful in the way I describe research and do my best to continue to present all sides.

These research issues tend to become very detailed, and I think we took readers about as far as they are willing to go, at least in one column. As I descend into senility, I would like to keep at least a couple of readers interested enough to hang around and send me e-mails to comfort me in my old age. I hope you will be one of them, and keep me honest as we try to make informed judgments about this confusing mass of information about schools that pours down on our heads every day.

Mark: I appreciate the opportunity to debate this issue with you, Jay. I'm sure you'll keep writing about (and promoting) AP. And I'll certainly be reading about it. Talk to you again soon.

© 2007 Washingtonpost.Newsweek Interactive