A couple of months ago someone told me the American Federation of Teachers was about to put out the first analysis of the nationwide performance of charter schools, based on the respected federal testing program, the National Assessment of Educational Progress (NAEP, pronounced to rhyme with tape).
It sounded like a pretty good story, something I should check out. But it was very nice day in August. I had a chance to see Joe Gibbs -- St. Joe in my family -- run the Redskins training camp out near Dulles airport. So I put off doing my job and was not happy to see the story the next day on the front page of the New York Times.
_____About the Author_____
Jay Mathews, a Washington Post education reporter, writes a weekly Class Struggle column exclusively for washingtonpost.com. He also covers school issues in a quarterly column for The Post Magazine. He can be reached via e-mail at email@example.com.
It said that the AFT analysis showed that kids in charter schools were not learning as much as kids in regular schools in their neighborhoods. This was big news, since advocates of charter schools -- public schools that operate independently of school district rules -- had said they were an improvement over the under-performing regular public school system.
Not everyone agreed with the AFT analysis, to say the least. There was a full-page newspaper ad by some scholars, denouncing the story and the report. There were bitter panel discussions. There was a lot of anger.
So I decided to wait until the venom subsided, and then try to shine some light on the issue by holding one of my occasional e-mail debates. I learn best when listening to smart people argue. I enjoy tossing a question to a couple of experts on opposite sides of a complicated issue and let them throw e-mails at each other for awhile. In this case, I was lucky to persuade F. Howard Nelson, co-author of the AFT study with Bella Rosenberg and Nancy Van Meter, and Martin R. West, research fellow at the Program on Education Policy and Governance at Harvard University, to present their very different views on the charter school data.
I will start the debate with a question for Nelson, and then get out of the way.
Jay Mathews: Please tell us the most important things you learned from your analysis of the NAEP data on charter schools, and what you think that means about what parents and students should think of such schools.
F. Howard Nelson: For parents, the NAEP (and lots of other) evidence indicates that charter schools are not a magic bullet and, if anything, the typical charter school is less effective than regular public schools with students from impoverished central city neighborhoods. Of course, NAEP does not report on individual schools, so it won't help individual parents and students with their individual decisions. But its overall message -- let the buyer beware -- should be heeded.
Often termed "the nation's report card," and widely reported in the press, NAEP has been testing the academic achievement of a nationally representative sample of students and publicly reporting the results since 1969. The 2003 administration of NAEP in math and reading in grades 4 and 8 represented the first time that a nationally representative sample of charter schools (grade 4) was included.
Compared to students in regular public schools, charter school students had lower achievement equal to about half a year of school. True, charter schools are slightly more likely to enroll poor students (54 percent compared to 46 percent), but when looking only at students eligible for free or reduced price lunch, the charter school disadvantage remained at about half a year of school. Central city charter schools scored at lower levels (statistically significant only in math) than regular public schools. Analyzed by race, regular public schools also did better, but the difference was within the margin of error. These findings are based on data available to the public and several researchers have replicated the results.
The publicly available NAEP data do not tell parents much about which attributes of charter schools (e.g., curriculum, governance, for-profits) work best. The administration continues to suppress the results of a detailed NAEP charter school questionnaire that could help determine what kinds of charter schools are most or least effective.
Martin R. West: The recent AFT study contains one accurate finding: Charter schools, on average, do not serve the privileged at the expense of the disadvantaged. Charter school students are more likely to be African American or Hispanic, to be eligible for a free or reduced price lunch, and to attend school in a central city.
As for how much students are learning in charter schools, the AFT study tells us nothing. Nor could it, simply because the NAEP provides information from only one point in time. To gauge student progress, one must know, at a minimum, where they begin in addition to where they end up.
Indeed, if Howard Nelson really believes that his analysis shows that "the typical charter school is less effective than regular public schools," then he must also believe that religious private schools are better than public schools. For the same NAEP data that Nelson relies upon for his charter study, when analyzed in exactly the same way, reveal that students in private schools outperform students in public schools.
But, of course, one cannot draw conclusions about how charter or private schools compare to traditional public schools, unless one compares two students with very similar backgrounds -- similar initial test scores, same race, similar parental education, similar place of residence, and more besides. Merely comparing groups of students sharing just one of these characteristics, as the AFT has done, proves little, because all these factors together affect student performance.
For Nelson to say his results have been replicated by others is dreadfully misleading. Of course, one can get the identical results if one analyzes the data in the same mindless manner. Mere repetition cannot make a wrong move right.
That the AFT would release -- and continue to defend -- a flawed study critical of charter schools is understandable. The AFT represents many who feel threatened by the charter schools spread. But it is precisely when groups with vested interests produce research that buyers need to beware. Wary consumers of this particular study will find only one conclusion that survives careful review: Charter schools are reaching a disadvantaged population.
Nelson: Mr. West's opinions reflect the ones he (with colleagues Paul Peterson and William Howell) espoused in a guest editorial in the Aug. 18 Wall Street Journal, published the day after the New York Times reported on the NAEP charter school data that the AFT unearthed. The gist of his argument is that because charter schools enroll more disadvantaged students than other public schools, they should be excused for performing poorly. But charter schools are only slightly more likely to serve poor students (in the NAEP data, 54 percent in charter schools compared to 46 percent in regular schools), and even after limiting our comparison to students eligible for free lunch, charter school students were still half a year behind regular public school students. Similarly, charter schools are concentrated in central cities, where there is a bigger marketplace for student recruitment, but after limiting our comparison only to central city schools, charter school students are still behind their peers in regular schools.
The AFT presented the 2003 NAEP results for charter schools, data that the administration had buried, without public announcement, in the Web-based NAEP Data Tool. Many of the more sophisticated analyses suggested by Mr. West are reasonable, but we did all the fair comparisons allowed by the limited NAEP Data Tool. Because the public, via journalists, ordinarily gets NAEP results in a highlights report, our presentation of the data essentially reproduced a typical NAEP highlights report. That's why it is important that others have exactly replicated our calculations. Contrary to our critic's distortions, we offered no conclusions about charter schools when we released the NAEP charter school data. (In answering the prior question asked by Jay Mathews, however, I think it is completely reasonable to conclude that that the NAEP data show that charter schools are not a magic bullet.)
Mr. West huffs and puffs about research methodology without saying a word about the administration's continuing suppression of public data (the NAEP Charter School Survey Questionnaire and the student-level data set usually available to researchers) that would allow all researchers, not just the AFT, to go beyond conventional NAEP reporting. Clearly, it is the NAEP results for charter schools that are at issue because they do not support the hyperbolic, ideological claims of many charter school advocates.
As he did with Peterson and Howell last August, Mr. West now raises the NAEP private school results in a version of the bait-and-switch strategy. For one, the private school results, unlike the suppressed charter school results, were published several months ago. Second, unlike our charter school report, they were presented without even the obvious adjustment for family income needed to make fair comparisons to public school students. How Mr. West can criticize our report and then tout these apples-to-oranges private school results is a bit illogical.
Mr. West's beef should be with NAEP, not our work. If he objects to the fact that NAEP is only a single point in time snapshot (a point that is also true of No Child Left Behind's adequate yearly progress formula), then he should stop using NAEP to tout private schools and indict regular public schools. If he thinks the family background information available through the NAEP Data Tool is too limited -- and we agree -- then he also should join us in our effort to have the 2003 student-level NAEP data set released to researchers and the still-suppressed results of the Charter School Survey Questionnaire made publicly available.
As a signer of the advertisement "[AFT] Charter School Evaluation Reported by The New York Times Fails to Meet Professional Standards," (Education Week 9/15/04), it is interesting that Mr. West does not criticize the recent charter school study by a fellow co-signer of the ad, Carolyn Hoxby. Like our presentation of the NAEP charter school findings, Hoxby's study used "information from only one point in time." Unlike our report, Hoxby presented no background comparisons, let alone comparisons of students with, "similar initial test scores, same race, similar parental education, similar place of residence, and more besides." Nor did Hoxby have her study "vetted by independent scholars" before it was released to the media. It is hard, then, to escape the conclusion that it is the poor NAEP charter school results we uncovered, and not our "methodology," that is the real issue for Mr. West.
West: Dr. Nelson's claim that he initially offered no conclusions about charter schools is disingenuous. "Charter schools underperforming" read the notice on the AFT Web site the day the organization released the study in August. Indeed, as of Oct. 19, it still does. And lest you think this was the work of an overzealous media office, I should also point out that the AFT's press release quoted Dr. Nelson himself saying that "[t]hese NAEP data reinforce years of independent research that show charter schools do no better and often underperform comparable, regular public schools."
In fact, the AFT study tells us next to nothing about the effectiveness of charter schools. The most defensible comparison the study makes is between charter school and public school students of the same race, if only because race is currently such a powerful predictor of student achievement. And as Dr. Nelson himself concedes, the results of this comparison show no statistically significant differences between charter schools and public schools.
But comparing students who share one characteristic in common is simply not enough to produce valid results. To give just one example, black students who attend charter schools may well come from poorer families than black students in traditional public schools, as most charter schools are located in central cities.
Charter school students may also differ from students in traditional public schools in ways that go beyond demographic characteristics. As schools of choice, charter schools are likely to be most attractive to families with students struggling in traditional public schools. Moreover, many charter schools explicitly target "at-risk" students. To blame these schools for low test scores would be like blaming an emergency room for a higher death rate than that of a podiatrist's office.
As Dr. Nelson points out, I am not alone in criticizing the AFT study's methodology and the media coverage that accompanied its initial release. An ad in the New York Times was signed by 30 other social scientists, including one Nobel laureate and many, like me, who have no vested interest in the charter school movement. All these scholars agreed that the AFT's study did not meet even the most basic professional research standards. As such, its conclusions about charter school performance should be ignored.
Dr. Nelson does not wish to believe that private schools are better than public schools. I have said explicitly that the NAEP provides no evidence for this. But Dr. Nelson cannot escape the fact that the very same methodology on which Dr. Nelson based his conclusion that charter schools are "if anything . . . less effective," when applied to a different set of schools, produces results that he and his AFT colleagues would be the first to criticize. Instead, he must now embrace them, if he is to be intellectually honest and internally consistent.
In any study of public, private, or charter schools, the findings depend upon the comparisons made. Using data from many more charter schools and making comparisons between charter schools and traditional public schools that are at least as defensible as those in the AFT study, Harvard University economist Caroline Hoxby finds large and positive differences between charter schools and traditional public schools nationwide. Curiously, though, Nelson is quick to disparage these findings, while touting his own.
Obviously, nailing down the truth (or truths) about charter schools will require considerably more information than either of these studies provides. Fortunately, research using more reliable methods to evaluate charter schools in specific cities and states is already underway. Some early results are favorable for charter schools; others are not.
Such variety is to be expected. Charter schools vary widely in their management and approach. Some are run by large management companies, others by lone entrepreneurs. Some are well established, others brand new. Some use standardized tests only when required by their state, others to organize their entire instructional program.
Yet every charter school, once established, should be held to the same high standards as other public schools. Fortunately, the No Child Left Behind Act's sanctions for schools needing improvement already apply in full to charter schools. Charter schools are also subject to regular review by their authorizers and, unlike traditional public schools, can be closed outright if their performance flags. And, above all, charter schools are accountable to the families who choose to use them.
Nelson: Mr. West (as he did with William Howell and Paul Peterson in the above referenced Wall Street Journal piece) seems to suggest that low NAEP scores in charter schools are caused by "poor, minority children." It sounds like the soft underbelly of racial discrimination caused by low expectations charter school. However, I find the low-expectations argument just ridiculous and ask only that advocates stop making it sound like urban charter schools are the only schools serving the disadvantaged.
The NAEP data reveal that black students in charter schools scored lower in math and reading than black students in other public schools, but not at statistically significant levels. Countless charter school advocacy groups bragged about this lack of statistical significance in one area (ignoring statistically significant lower achievement for low-income students in charter schools), as if it demonstrated that charter schools are superior to other public schools for minority children. At best, it suggests only that charter schools have failed to raise achievement for minority students at a faster rate than public schools. I disagree that race alone is a powerful predictor of student achievement. After accounting for differences in economic and social circumstances, race barely has anything to do with student achievement.
Referring to the Hoxby study, I agree with Mr. West when he says that "Obviously nailing down the truth (or truths) about charter schools will require considerably more information than either of these studies." It is a far fairer conclusion than the ones drawn by the numerous charter school advocates who continue to trash the NAEP charter school data, because they don't like the results, and tout her study because they do like the results. In my previous comments, I pointed out that signers of the New York Time advertisement criticizing our study should be equally critical of the Hoxby study for the same (or more) shortcomings, but I did not disparage her findings.
Now, however, I would like to list a few problems with the District of Columbia results in her study (based on data that she has shared) that raise serious question about quality and accuracy in other states in the study:
Hoxby claimed that 100 percent of charter schools were studied, but only the nine charter schools authorized by the Public Charter School Board were studied while several others (those authorized by the school district) were not included.
Charter school proficiency was measured by NCLB proficiency standards (40th percentile) while the comparison school achievement was measured by the much higher proficiency standard used by the test maker (SAT-9).
The combined results of the two large schools managed by the Edison Schools Inc., located in different parts of the city were matched to only one neighborhood school.
The nearest neighborhood schools in the Hoxby study almost always differed from the nearest school selected by greatschools.net or mapquest.com (frequently, several nearer schools were found with higher achievement).
Even when using Hoxby's lower-scoring comparison schools, merely adopting a common definition of proficiency -- the one used for NCLB purposes -- wipes out most of the 35 percent difference in proficiency favoring charter schools. Thus, both methods (NAEP and Hoxby's) yielded the same results for the District of Columbia -- no charter school effect. And these findings are better than those found in an older study by Henig, Hoyoke Lacreno-Paquet and Moser who concluded that even after controlling statistically for differences in socioeconomic status, more charter schools scored at the "below basic" skill level than students in other District of Columbia public schools.
There are many things that Mr. West and I agree upon besides the fact that neither our study nor Hoxby's study settles the charter school debate. For example, in any study of public, private or charter schools, the findings depend upon the comparisons. We made all of the fair comparisons we could using the limited NAEP Data Tool and look to the more sophisticated analyses that will be possible if the student-level NAEP data are released to the public. We expect that student-level data from the 2003 private school comparisons will show -- as in past NAEP data collections -- no difference between private and public schools after accounting for parental education and course taking.
In many ways, the release of the NAEP charter school findings and the ensuing debate has not addressed many important questions. Do charter schools have anything new to offer? Or, is it all about opening public schools to profit-making? Why aren't charter schools subject to rigorous evaluation (that is, rigorous testing and identification of new methods for reaching low-achieving kids)? For example, there is still no rigorous evaluation of the D.C. charter school experiment so that the practices of successful charter schools can be transferred to the regular system. After all, freedom for freedom's sake is not reason enough to establish a system of charter schools.
West: To repeat, the comparisons Dr. Nelson and his colleagues made using the "limited NAEP data tool" are hardly "fair." They rely on achievement data from only one point in time, control for differences in students' backgrounds in the crudest manner imaginable, and make no attempt whatsoever to account for circumstances that might lead students to seek an alternative to their traditional public schools. In each of these respects, they violate basic research standards.
Rather than defend his own study (how could he?), Dr. Nelson now accuses me and my colleagues of racial discrimination for having pointed out its flaws. This is preposterous. Any credible effort to gauge schools' effectiveness must account for the ethnic composition (among many other characteristics) of their student bodies. Indeed, were Dr. Nelson to claim otherwise in a typical undergraduate research seminar (much less at a professional conference), he would be laughed out of the room.
Dr. Nelson's assertion that race "barely has anything to do with" student achievement after accounting for differences in students' social and economic backgrounds is simply false. A study published this year by economists Steven Levitt of the University of Chicago and Roland Fryer of Harvard's Society of Fellows demonstrates that substantial racial gaps in achievement unexplained by students' background characteristics emerge as early as the first grade.
This is regrettable. It is also true. Dr. Nelson's unwillingness to recognize the historical legacy of racism in America and the abiding relevance of race as a predictor of students' academic success is both naive and self-serving.
Dr. Nelson uses additional space he might have spent defending his own study to raise doubts about Caroline Hoxby's recent analysis of charter schools. Specifically, Dr. Nelson mentions findings from what appears to be an ongoing secondary analysis of the data Hoxby compiled for a single city -- a city that includes only 1% of the charter schools included in her study.
For readers unfamiliar with this research, Professor Hoxby compared the proficiency levels of virtually every charter school in the nation to their nearest traditional public school (and, where appropriate, the nearest school with a similar racial composition). She found that students in charter schools nationwide are more likely to be proficient in both math and reading, especially in those states where charter schools are well established (for more information, see her study) .
Although imperfect, using geography to identify a comparison group for charter schools has obvious advantages over the AFT's approach, which was to compare students in charter schools to a national public school sample that inevitably included students from wealthy suburbs and even from states without any charter schools. Moreover, Professor Hoxby is currently conducting research that exploits the lottery-based admissions policies of oversubscribed charter schools to gauge their effectiveness -- probably the only way to generate definitive results. In contrast, the AFT's study seems like a drive-by shooting.
What general lessons should we draw from the AFT's study and the events surrounding its release? Two seem especially important:
1. In contentious debates over public policy, bad information is worse than none at all. By dredging up misleading information and presenting it out of context, the AFT set back ongoing efforts to improve the quality of evidence on which education policy decisions are made. And by continuing to promote a hopelessly flawed study, the organization has forfeited whatever status it may have had as a credible voice in public discussions of how to improve America's schools. There can no longer be any doubt that the material interests of AFT members take precedence over the interests of the children they teach.
2. While the statistics on charter schools from the 2003 NAEP are not at all useful for assessing school effectiveness, they do offer, for the first time, a glimpse of the makeup of a nationally representative sample of the students who attend them. The evidence is striking: Roughly 62 percent of 4th graders in the NAEP charter-school sample attend a school located in a central city, compared with just 32 percent of NAEP 4th graders in traditional public schools. Roughly 33 percent of the charter school students are African American, compared with only 18 percent of the public school students. Fifty-four percent of charter school students qualify for free or reduced lunch programs, compared with 46 percent of public school students.
Given the locations in which states and districts typically allow charter schools to open and the characteristics of families most eager for alternatives to traditional public schools, these differences can hardly come as a surprise. For the foreseeable future, charter schools are likely to serve high concentrations of underprivileged students. What remains unclear is how much they can do for them. Sadly, the AFT study tells us nothing about that.
Nelson can be reached by e-mail at firstname.lastname@example.org.
West can be reached by e-mail at email@example.com.