New Studies Say AP Works

But Saul Geiser, the University of California scholar who produced his own massive AP and college success study in 2004, said these comparisons are still a problem.

"The main technical issue that bedevils not only the Texas studies but all research of this kind concerns the question of selection bias: To what extent is AP students' success in college a reflection of the AP program itself (a 'program effect') as opposed to the personal characteristics of the students selected for the program (a 'selection effect')?" he asked. "This question is particularly important in evaluating AP because the program historically has targeted 'better,' academically stronger students who could be expected to perform well in college even without AP.

"Both of the Texas studies attempt to address the problem of selection bias by 'matching' AP and non-AP students on other measures, primarily SAT scores, in order to control for student background characteristics. But neither study controls for a sufficiently robust set of student characteristics to rule out the hypothesis that selection effects account for most of the relationship between AP and college outcomes. The larger study of all Texas public institutions by Hargrove, Godin and Dodd matched AP and non-AP students only on two measures, SAT I scores and participation in free-lunch programs, while the smaller study matched students only on SAT I scores and high-school class rank. As a result, the studies control for far fewer variables than previous researchers have considered, leaving open the possibility, and indeed probability, that systematic differences in student motivation, academic preparation, family background and high-school quality account for much of the observed difference in college outcomes between AP and non-AP students."

2. How AP affects success in the next college course

This is the part of the second Texas study, by University of Texas-Austin researchers Leslie Keng and Barbara Dodd, that attempts to shed light in the so far clueless debate over whether AP and IB students deserve college credit and a chance to skip to a more advanced college course.

Many selective schools allow only students with the highest grade, a 5, on the AP exam to get credit. Some insist that their introductory courses are so special that no AP grade is high enough. But the amount of research they have devoted to proving their point is laughingly small. Harvard made a rule that only 5s will count -- and only if a student is going for sophomore standing -- based on one study of three courses that showed AP students not doing as well as students who first took the college's intro course. A few colleges have data from a similarly small number of courses that show the AP or IB students do better on the advanced course than students who have taken the college's intro course. But most colleges have no data. When you ask them on what they rest their restrictive rules, they shrug and say the faculty wanted it that way.

The Texas study has to be read very carefully, because although it concludes that students who received AP credit in 10 subjects at UT did better in the advanced course than students who first took the intro course, picking the right advanced course for the researchers to gather data from and study was difficult, and in some cases, with some categories of student, the triumph of the AP kids is not so clear.

3. The worthiness of AP science

In my Nov. 28 column, I discussed in detail a yet unpublished study by Harvard researcher Philip M. Sadler and University of Virginia researcher Robert H. Tai. They surveyed a relatively small sample of AP science students at 63 colleges and concluded that their AP experience did not help them much, if at all, in similar college introductory courses. They sadly did not look at AP students who took advanced college courses, so their work can not be directly compared to the UT study, but theirs and the Texas study are going to be the principal pieces of evidence in debates on this issue.

In the second Texas study, Sadler found some of those same patterns among the AP science students.

"I note that in the Dodd and Keng study that non-AP students (taking the prerequisite course in college biology) get better grades than AP-Credit students in the subsequent (Bio 303 ) courses. I would like to see their chemistry data on subsequent course grade, which does not appear to be included. This can be interpreted as AP courses not being the equivalent of taking the university's biology prerequisite to second year biology. . . . This all may differ by subject matter area.

"I also worry about the methodology of matching rank and SAT/ACT scores. It is unclear whether high school rank was done with AP weighting or not, since this will impact rank. I would much rather use actual course grades in relevant subjects and actual SAT/ACT scores in regression models. I remain unconvinced that the study controlled for enough variables to rule out alternative hypotheses to that of AP being responsible for better performance seen in GPA and graduation rates. I think it is best to control for performance and level of high school math, performance in prerequisite courses to AP in HS, parental education, community affluence, etc., if possible."