The Washington Post

Researchers fault L.A. Times methods in analysis of California teachers

University of Colorado researchers reported Monday that there were potential weaknesses in methods the Los Angeles Times used last year to rate elementary school teachers for a series that stoked national debate in a key arena of education reform.

The newspaper said it stood by its investigation.

At issue are evaluation techniques that estimate, through analysis of standardized test scores, how much a given teacher helps or hinders the academic growth of students. Such methods are generally supported by the Obama administration but controversial in the education world.

These "value-added" models, as they are known, have been embraced in the D.C. public schools and elsewhere by officials who say they are improving evaluation systems that otherwise fail to account for what matters most: student achievement.

In many places, the models are beginning to influence decisions about teacher pay and tenure. That means they are a central element of reforms related to President Obama's assertion in his State of the Union address that good teachers should be rewarded and bad teachers held accountable.

The Times series broke ground in part because it was based on an unusually detailed set of testing data obtained from the Los Angeles Unified School District through a public records request. The newspaper paid an economist to run a value-added analysis of the numbers. Then it published a database that named 6,000 teachers and rated their effectiveness in reading and math.

The Colorado researchers, who were funded by a policy center with some backing from teacher unions, obtained from the district what was believed to be the same raw data the newspaper used. The researchers concluded that the Times relied on a methodology that omitted factors that might have changed many teacher ratings. These included the influence of peers in a classroom, the history of a student's test performance and the overall school demographics.

An alternative analysis that included such factors, the researchers said, might have changed how roughly half of a set of 3,300 fifth-grade teachers in Los Angeles were evaluated in reading.

"It may well be the case that all value-added models are flawed, but some are more flawed than others," said Derek Briggs, the lead researcher. "One of the things we show in our report is that the choice of what 'control' variables are included or excluded from the model can matter a great deal to inferences about teacher effectiveness."

The study was released by the National Education Policy Center at the University of Colorado at Boulder. Among the policy center's financial backers are the Ford Foundation and the Great Lakes Center for Education Research and Practice. The Great Lakes Center's membership includes affiliates of the National Education Association, the largest teachers union. Local and national union leaders have criticized the Times project as unfair to teachers.

David Lauter, an assistant managing editor for the Times, said that the newspaper stands by its reporting and that some of the study's findings corroborated the Times investigation. Lauter said the Times remains confident in the work of the economist, Richard Buddin. Buddin works for Rand Corp., based in Santa Monica, Calif., but Rand was not involved in the analysis.

Lauter said Briggs and the study's co-author, Ben Domingue, appeared to confuse Buddin's technical analysis with separate decisions the Times made about rankings and publication. He called that "a pretty big flaw." Briggs denied any such mixup.

Still, Lauter said he welcomed the critique.

"Part of the whole point in our putting all this effort into this work was to spark a public debate about how best to evaluate teachers," Lauter said. "The more that people get into this, the better."

The back and forth on the Los Angeles ratings underscores that many practical questions remain about value-added analysis even as its influence grows. Among them: How to judge teachers whose students do not take standardized tests in reading and math? How to account for class placement and team instruction? How to account for the many advantages of affluent students and the many disadvantages of the poor?

"There's not necessarily a right answer as to what model you want to adopt," said Dan Goldhaber, director of the Center for Education Data and Research at the University of Washington at Bothell, who reviewed the Briggs study. He said value-added analysis can be useful as long as policymakers acknowledge "there's a lot of nuance" in such estimates.

"What Derek has shown is that when you control for different sets of variables, the estimates vary," said Thomas Kane, a Harvard University professor of education and economics. He is studying teacher effectiveness for the Bill and Melinda Gates Foundation. "But we still don't know yet which [model] was the right one and how far off from the truth the various estimates are."

Nick Anderson covers higher education for The Washington Post. He has been a writer and editor at The Post since 2005.


Success! Check your inbox for details. You might also like:

Please enter a valid email address

See all newsletters

Show Comments

Sign up for email updates from the "Confronting the Caliphate" series.

You have signed up for the "Confronting the Caliphate" series.

Thank you for signing up
You'll receive e-mail when new stories are published in this series.
Most Read


Success! Check your inbox for details.

See all newsletters

Your Three. Videos curated for you.
Play Videos
Deaf banjo player teaches thousands
Unconventional warfare with a side of ale
It's in the details: Five ways to enhance your kitchen makeover
Play Videos
Drawing as an act of defiance
A fighter pilot helmet with 360 degrees of sky
Border collies: A 'mouse trap' for geese on the National Mall
Play Videos
Bao: The signature dish of San Francisco
This man's job is binge-watching for Netflix
What you need to know about Planned Parenthood
Play Videos
How to save and spend money at college
Pandas, from birth to milk to mom
Europe's migrant crisis, explained

To keep reading, please enter your email address.

You’ll also receive from The Washington Post:
  • A free 6-week digital subscription
  • Our daily newsletter in your inbox

Please enter a valid email address

I have read and agree to the Terms of Service and Privacy Policy.

Please indicate agreement.

Thank you.

Check your inbox. We’ve sent an email explaining how to set up an account and activate your free digital subscription.