University of Colorado researchers reported Monday there were potential weaknesses in methods the Los Angeles Times used last year to rate elementary teachers for a series that stoked national debate in a key arena of school reform.
The newspaper said it stood by its investigation.
At issue are evaluation techniques that estimate through analysis of standardized test scores how much a given teacher helps or hinders the academic growth of students. Such methods are generally supported by the Obama administration but controversial in the education world.
These "value-added" models, as they are known, have been embraced in D.C. public schools and elsewhere by officials who say they are improving evaluation systems that otherwise fail to account for what matters most: student achievement.
In many places, the models are beginning to influence decisions about teacher pay and tenure. That means they are a central element of reforms related to President Obama's assertion in his State of the Union address that good teachers should be rewarded and bad teachers held accountable.
The Times series broke ground in part because it was based on an unusually detailed set of testing data obtained from the Los Angeles Unified School District through a public records request. The newspaper paid an economist to run a value-added analysis of the numbers. Then it published a database that named 6,000 teachers and rated their effectiveness in reading and math.
The Colorado researchers, who were funded by a policy center with some backing from teachers unions, obtained from the district what was believed to be the same raw data the newspaper used. The researchers concluded that the Times relied on a methodology that omitted factors that might have changed many teacher ratings. These included the influence of peers in a classroom, the history of a student's test performance and the overall school demographics.
An alternative analysis that included such factors, the researchers said, might have changed how roughly half of a set of 3,300 fifth-grade teachers in Los Angeles were evaluated in reading.
"It may well be the case that all value-added models are flawed, but some are more flawed than others," said Derek Briggs, the lead researcher. "One of the things we show in our report is that the choice of what 'control' variables are included or excluded from the model can matter a great deal to inferences about teacher effectiveness."
The study was released by the National Education Policy Center at the University of Colorado at Boulder. Among the policy center's financial backers are the Ford Foundation and the Great Lakes Center for Education Research and Practice. The Great Lakes Center's membership includes affiliates of the National Education Association, the largest teachers union. Local and national union leaders have criticized the Times project as unfair to teachers.
David Lauter, an assistant managing editor for the Times, said that the newspaper stands by its reporting and that some of the study's findings corroborated the Times investigation. Lauter said the Times remains confident in the work of the economist, Richard Buddin. Buddin works for Rand Corp., based in Santa Monica, but Rand was not involved in the analysis.
Lauter said Briggs and the study's co-author, Ben Domingue, appeared to confuse Buddin's technical analysis with separate decisions the Times made about rankings and publication. He called that "a pretty big flaw." Briggs denied any such mixup.
Still, Lauter said he welcomed the critique.
"Part of the whole point in our putting all this effort into this work was to spark a public debate about how best to evaluate teachers," Lauter said. "The more that people get into this, the better."
The back and forth on the Los Angeles ratings underscores that many practical questions remain about value-added analysis even as its influence grows. Among them: How to judge teachers whose students do not take standardized tests in reading and math? How to account for class placement and team instruction? How to account for the many advantages of affluent students and the many disadvantages of the poor?
"There's not necessarily a right answer as to what model you want to adopt," said Dan Goldhaber, director of the Center for Education Data and Research at the University of Washington-Bothell, who reviewed the Briggs study. He said that value-added analysis can be useful as long as policymakers acknowledge that "there's a lot of nuance" in such estimates.
"What Derek has shown is that when you control for different sets of variables, the estimates vary," said Thomas Kane, a Harvard University professor of education and economics. He is studying teacher effectiveness for the Bill and Melinda Gates Foundation. "But we still don't know yet which [model] was the right one and how far off from the truth the various estimates are."