I am emulating my favorite fourth grader, Bart Simpson, and writing on the blackboard something many teachers have been telling me I need to remember:
Correlation Is Not Causation
_____About the Author_____
Jay Mathews, a Washington Post education reporter, writes a weekly Class Struggle column exclusively for washingtonpost.com. He also covers school issues in a quarterly column for The Post Magazine. He can be reached via e-mail at email@example.com.
Correlation Is Not Causation
Correlation Is Not Causation
Correlation Is Not Causation
I am wearing out several pieces of chalk because in my Nov. 23 column I failed to warn readers that a new Texas study indicating a connection between taking Advanced Placement (AP) tests in high school and graduating from college was not proof that one caused the other. This is the first lesson in any statistics course, and I am aware of it, but I tend to forget it when I get excited. Many kind readers, most of them teachers, reminded me of this in time for me to correct my error when I did a Dec. 5 version of that column for
The Post's Outlook section. The Texas study provided strong and valuable new evidence for the power of AP, but I was wrong to say it was conclusive.
Now I have another intriguing piece of educational research in front of me, and I don't make the same mistake twice. Saul Geiser and Veronica Santelices of the University of California at Berkeley have just released a preliminary version of a study I think is as important to our understanding of college-level courses in high school as the Texas study I described last month. I will try to discuss what they have found without leaping to conclusions, particularly since a report this rich and complicated is almost certain to be misinterpreted.
Here is a link to a pre-publication draft of the study, "The Role of Advanced Placement and Honors Courses in College Admissions," which is now being peer-reviewed. Here is the news story I wrote about it last week. For those who don't have time to read the whole thing, this quote from the report sums it up well:
"The main finding . . . is that, controlling for other academic and socioeconomic factors, the number of AP and honors courses taken in high school bears little or no relationship to students' later performance in college. The study is based on a sample of 81,445 freshmen entering the University of California (UC) [including eight campuses] between 1998 and 2001. While student performance on AP examinations is strongly related to college performance, many students who take AP courses do not complete the associated AP exams, and merely taking AP or other honors-level courses in high school is not a valid indicator of the likelihood that students will perform well in college. If the UC sample is representative of other selective colleges and universities, those institutions may need to reconsider the manner in which such courses are treated in 'high stakes' admissions. Such reconsideration assumes special importance in view of the marked disparity in access to AP and honors courses among disadvantaged and underrepresented minority students."
This column attempts to be a full-service provider on a wide range of school issues. But it is clear that my deepest interests are in two areas, (1) improving schools for average and below average students, particularly those from low-income families, and (2) demystifying and improving the college-admissions process. The Geiser-Santelices report addresses both of those concerns, which is why I am so eager to write about it. But it requires careful reading, and I suspect many are going to stumble as I did before all of its points become clear.
I wish, for instance, that I had made clear in my news story that the peculiar nature of the sample of students used by Geiser and Santelices makes their research less helpful in answering the question dealt with in the earlier Texas report -- can taking an AP course and test in high school improve a student's chances of graduating from college, even with an average student who struggles with the course and the exam?
The question is important because it focuses on the majority of American high school students, those B and C students who are often placed in mediocre classes and not encouraged to take AP or International Baccalaureate (IB) courses and tests that many experts say improve their chances of surviving in college. I invented the Challenge Index, a way of rating high schools by AP and IB test participation, in order to encourage more schools to open those courses to everyone and to make sure those students took the final exams.
In one way, the Geiser-Santelices study seems to support that effort, for it says that taking an AP or IB course without taking the final exam does not predict academic performance in college, while taking and doing well on an AP or IB exam does. In another way, the study seems to contradict the notion that just trying to do well on the test has a benefit in college, because those with poor AP or IB exam grades do not appear to be any more likely to do well in college than those who have taken the courses but not the tests.
What I failed to say in my news story is that we cannot learn much from this study about the effect of AP and IB on the chances of average or below-average high school students graduating from college because there are almost no such students in the Geiser-Santelices sample. UC campuses accept only the top 12 percent of California high school graduates, the A and high-B students who are not the least bit average. Those students have already gotten, in many cases, good doses of the college trauma necessary for them to graduate when they get to the university of their choice, and many are skilled and motivated enough to get through college even without that extra high school preparation.
But as Geiser and Santelices make clear, there are an enormous number of average students who are not in their study, but who want to go to college, need that high school preparation and are, nonetheless, denied even a taste of those challenging courses. The study looked at the 117,650 college-bound seniors who took the SAT I test in California in 2002 and found that 54.9 percent of them, a total of 64,577 kids eager to get to college and graduate, had been given no AP, IB or honors courses -- that's right, zero, nada, none -- either because such courses were not offered by their poverty-stricken high schools or because they were tracked out of such courses by their clueless high schools. Mike Riley, superintendent of the Bellevue, Wash., schools, called this fact "perhaps the most significant in the study. Kids heading to college without a college-prep high school program are headed for failure."
The earlier studies that show a positive college graduation effect from taking such courses in high school, such as the aforementioned Texas study by the National Center for Educational Accountability and the 1999 study by U.S. Education Department researcher Clifford Adelman, include average students. Adelman's study looked at nearly 1,000 four-year colleges of all degrees of selectivity, and it included students who had transferred from community colleges.
"We cannot construct or execute policy for everyone on the basis of the performance and history of elites," Adelman told me. If the UC researchers "did the same study with California State University students -- as opposed to UC students -- and include the community college transfers, I doubt they would have reached the same conclusions" about the benefits of intense academic experiences in high school.
Geiser and Santelices are, in any case, less interested in AP's and IB's effect on learning in high school than they are on the college-admissions issue. Their work is likely to spark a national debate over the emphasis that selective colleges place on AP and IB, and the extra grade points that many school districts give students who take these courses. At the very least, the UC undergraduate admissions system will have to rethink its 22-year-old policy of adding a full grade point, making every A worth 5 points rather than just 4, to any grade in any course labeled college-level or honors, even if no college-level exam is given to verify its rigor.
Bob Schaeffer, public education director for FairTest, the National Center for Fair & Open Testing, congratulated Geiser and Santelices for challenging the widespread assumption that those without AP, IB or honors courses on their transcripts are not as good as applicants who have such credentials. "Information from an applicant's high school record should be used only for the purposes for which it has been validated," Schaeffer said. "AP courses and exams have never been validated as admissions tools. They were designed solely for placement purposes."
Trevor Packer, executive director of the College Board's AP program, applauded the UC study for much the same reason, while making a different point. The study, he noted, "does not question or critique" the traditional use of AP "as a tool for justifying the placement of students into sequential college courses and/or the granting of college credit."
Packer said he would like the UC researchers to study the effects of different sorts of AP classrooms on college success. He said he thought that classrooms that insist on students tackling college-level material and taking the AP exam produce better effects on college graduation than classrooms where the AP label has been slapped on a mediocre course that shuns AP exams but gets its students that extra grade point to make their UC applications more impressive.
Packer said he also would like the UC researchers to refine the separation in the study between students who took AP exams and those who did not, and make sure transfers are not counted as college drop-outs in the analysis of college success. Packer and Riley both said they thought the study's conclusion that AP-inflated grades don't predict college grades as well as non-inflated grades may be in part a truism. But they also praised the depth of the UC research and said they looked forward to the final version after peer review.
There remains, however, the problem of how the report will be interpreted by the public. Geiser and Santelices examined several options for changing the way AP, IB and honors courses are used as admissions criteria at selective institutions. These options, such as reducing but not eliminating the current weight given to AP courses, attempt to minimize the adverse impact on admissions of disadvantaged and underrepresented students while at the same time preserving the incentive for students to attempt rigorous coursework in high school. The report makes no recommendations but concludes that there seems to be no perfect solution that fully addresses both of these concerns and that might be equally appropriate for all institutions. It is obviously an advantage to colleges to encourage students to take more challenging high school courses so that they are better prepared as they enter college, but finding a fair evaluation of those classes may prove difficult.
Riley, who has made AP and IB a priority in his district, said he agreed with the researchers that low-income and minority students do not have enough opportunities to take college-level courses and tests in high school, but he said the solution should NOT be to devalue AP courses. Instead, he said, "the United States should make preparation for and access to and success in AP classes the number one goal for all students, but most especially those traditionally underserved by its public schools."
There will be much more discussion of the statistical complexities in the Geiser-Santelices report. I am paying as close attention as I can and trying to grasp the technical issues. But I will, as usual, be most influenced by the people who tell me how these issues have affected their own lives.
The mother of an average student in a Maryland public high school with many low-income students wrote me last week to say "my son is one of those just-barely-B students who's actually getting an A in his AP class (which he didn't want to take) because it's focused, challenging and has a great teacher. He would never have taken it if he weren't pushed, because he's become convinced over time that he and his classmates are just not very smart -- a result of years of being told his school scored lowest."
Any study that helps more students like that is fine with me, no matter how long it takes me to understand exactly how that very intricate and vital process actually works.