The following post was written by two members of a committee that authored a report released last month by the National Research Council on evaluating D.C. public school reform under Michelle Rhee. They are Robert Hauser, executive director of the Division of Behavioral and Social Sciences and Education at the National Research Council, and Christopher Edley, dean of the School of Law at the University of California, Berkeley.
The report, called “A Plan for Evaluating the District of Columbia’s Public Schools: From Impressions to Evidence ,” concluded that the standardized test scores that have been trumpeted to show improvement in the D.C. schools under Rhee provide limited information about the causes of improvements or variability in student performance.
The report came under criticism recently in an opinion piece written by Paul E. Peterson, director of Harvard’s Program on Education Policy and Governance, in a report that challenges the “case” against Rhee and can be found on the website of the publication Education Next.
The National Research Council report is the first in a series of mandatory reports under the law that allowed then-Mayor Adrian Fenty to take control of the city’s public schools and unilaterally appoint Rhee as chancellor. (She left the job last October after about 3 1/3 years.)The National Research Council is the research arm of the National Academies, which include the National Academy of Sciences, the National Academy of Engineering and the Institute of Medicine.
Peterson, a fan of Rhee’s, takes issue with this report and a separate one that was more critical of Rhee’s tenure, accusing the authors of mischaracterizing test score gains, going beyond their mission and making other errors. Here Hauser and Edley write in response.
By Robert Hauser and Christopher Edley
Washingtonians are accustomed to seeing their elected and appointed leaders attempt to improve public education in the city: Since 1804 there have been 17 different school governance and administrative regimes, the most recent established in 2007. That legislation transferred authority from an elected board to the Mayor, called for fundamental changes to management structures, created a public school charter board, and, perhaps most noticeably, allowed the mayor to recruit a new school chancellor. This “jolt” to the system was clearly motivated by widespread dissatisfaction with the quality of teaching and learning, and it came with a requirement that the results of the reforms be independently evaluated.
The National Research Council of the National Academy of Sciences was asked to take on that important challenge and has now released the first of its reports, A Plan for Evaluating the District of Columbia’s Public Schools: From Impressions to Evidence. The prepublication version of the report is freely available at http://www.nap.edu/ . Not surprisingly, the report has garnered both respect and criticism (for example, see the April 12 Washington Times article by the Harvard political scientist Paul Peterson, based on a longer article that will soon appear in another magazine he edits and is now on the website).
In light of claims made about both the purpose and results of the National Research Council report, it is important to clarify its main findings and recommendations. The report’s authoring committee – which we co-chaired – was asked to develop a plan for evaluating the District’s public schools. The committee included 15 nationally recognized educational researchers and practitioners, a number of whom live in Washington D.C. and have deep knowledge of the city and its schools, and its report was independently reviewed by 13 more experts before revision and public release.
We recommended that there be a sustained, independent, and scientifically rigorous examination of a wide range of data, using many types of analysis, to provide a comprehensive picture of how the District as a whole is functioning and its progress over time that will help all stakeholders – D.C. officials, school teachers and administrators, parents, students, and other citizens – to improve the D.C. schools.
Although it may be disappointing to people who were hoping for a “thumbs-up or thumbs-down” appraisal of former Chancellor Michelle Rhee’s performance, we were not asked to evaluate her performance or the specific effects of her tenure on the schools, and we did not do so.
Our recommendations were motivated largely by our review of publicly available data, which are incomplete and inadequate to support definitive conclusions about the effects of the reform initiative or the functioning of the school system.
For example, although there is evidence of test score gains, our committee cautioned against premature causal explanations or policy advice based on impressionistic evidence. We explained why test scores alone cannot answer essential questions about the reforms or about how effectively the District is carrying out its many responsibilities. This theme was echoed by the former chancellor, who said in a recent blog post that “rising test scores are a critical measure of school progress, but they aren’t the only metrics we can use....”
Our report is cautious about comparisons between D.C. and other cities, as is appropriate in a scientific analysis of a complex set of issues. Thus, we considered the role of chance in findings from sample data, like those from the National Assessment of Educational Progress (NAEP). When compared with the other school districts assessed by NAEP in 2007 and 2009, D.C. showed an increase that is reliably higher than that of only two districts (Austin and Cleveland) in grade 4 mathematics and one district (Cleveland) in grade 4 reading – but no others. This finding has no particular ideological or political bent; it is the result of careful and straightforward analysis of the data.
We were also careful to follow an elementary tenet of scientific inquiry: that correlation is not evidence of causation. Thus, although our report notes the encouraging news that truancy has declined in D.C., we did not infer a cause-and-effect relationship between that change and academic achievement.
We believe our report lays the groundwork for a rigorous and credible program of evaluation and research, one that would bring to the problems of education in DC the very best available expertise and reasoned judgment. The city’s children deserve nothing less.
Follow The Answer Sheet every day by bookmarking http://www.washingtonpost.com/blogs/answer-sheet. And for admissions advice, college news and links to campus papers, please check out our Higher Education page. Bookmark it!