The panel, composed of 10 researchers from across the country, met Friday in Washington. The public portion of its day-long meeting was devoted to discussing how firmly test scores can be trusted in light of security concerns.
It’s an issue the panel will have to confront as it uses test scores to draw conclusions about D.C. students’ academic performance during an era of closely watched reforms. The group’s broad evaluation, required by the same 2007 legislation that brought the city mayoral control of the schools, is due in September 2014.
“Our evaluation is so dependent on the test score data,” said Lorraine McDonnell, a co-chair of the panel and a professor of political science at the University of California at Santa Barbara. “We see this as an issue of trying to ascertain data quality.”
Suspicions that cheating skewed D.C. schools’ test results have simmered since 2011, when USA Today published an investigation showing more than 100 schools with unusually high numbers of wrong-to-right erasures on answer sheets.
Multiple subsequent investigations turned up no evidence of widespread cheating. Still, there are lingering concerns that haven’t been explained, such as schools that made rapid gains until test security was tightened and their scores plummeted.
In an effort to better understand the security concerns, the NRC panel invited Carswell Whitehead, a representative from the test developer ETS, to explain statistical tools that can be used to flag possible irregularities. Whitehead also described precautions that school systems can use to prevent cheating.
There was little discussion about the District’s test-security protocols. But several panel members raised concerns about rumors that there are no time limits on the city’s standardized tests — which would put schools at a higher risk for cheating, according to Whitehead.
A testing guide that the Office of the State Superintendent for Education published in 2011 specifies that there are “suggested” and not “fixed” time limits. But Chancellor Kaya Henderson, who attended part of the two-hour session, told the panel that tests are always timed.
The panel also invited Atlanta Journal Constitution reporter Heather Vogell to recount cheating probes in that city. Vogell described how the Georgia Bureau of Investigation, armed with subpoena power, turned up evidence that nearly 200 Atlanta teachers and principals cheated on 2009 state tests at dozens of schools.
State and local authorities’ multiple previous investigations into the same allegations had failed, Vogell said, because they were hobbled by lack of resources and lack of independence from school leaders.
McDonnell declined to comment on the panel’s discussions about test security. She said the final report may include recommendations for tightening security in the future.
The panel is working in conjunction with the DC Education Consortium on Research and Evaluation (DC-EdCORE), a group of local experts led by researchers at the graduate school for education at George Washington University.
DC-EdCORE will publish a series of analyses of D.C. schools data that the panel will use to inform its final evaluation. The first of those analyses is scheduled to be published in April.