The Washington PostDemocracy Dies in Darkness

Errors found in D.C. teacher evaluations (2nd update)

(2nd Update: details on affected teachers)

More than 40 teachers in D.C. public schools received incorrect evaluations for 2012-2013 because of errors in the way the scores were calculated and one was fired as a result.

The president of the Washington Teachers’  Union, Elizabeth A. Davis, has asked for details from D.C. Schools Chancellor Kaya Henderson in a letter (text below) that says that the problems were found by Mathematica Policy Research, a partner of the school system’s. The mistakes were found in the individual “value added” scores for teachers, which are calculated through a complicated formula that includes student standardized test scores.

This “VAM” formula is part of the evaluation system called IMPACT, begun under former chancellor Michelle Rhee in 2009. Henderson, Rhee’s successor, continued with IMPACT, though this year she reduced the amount of weight given to test scores from a mandatory 50 percent to at least 35 percent. (See below for IMPACT chart).

Testing experts have long warned that using test scores to evaluate teachers is a bad idea, and that these formulas are subject to error, but such evaluation has become a central part of modern school reform. In the District, the evaluation of adults in the school system by test scores included everybody in a school building; until this year, that even included custodians. In some places around the country, teachers received evaluations based on test scores of students they never had. (It sounds incredible but it’s true.)

My colleague Nick Anderson reported in this story that 44 teachers were involved, and one was fired as a result of an evaluation that was too low. Half of the evaluations for 44 teachers were too high and half too low, according to Jason Kamras, chief of human capital for the school district. The teachers with bad evaluations represent about 1 percent of the system’s 4,000 teachers but nearly 10 percent of those whose evaluations are based in part on student standardized test scores. Kamras said the fired teacher will be reinstated with back pay, that those teachers with ratings that were too high will not be lowered but those with too-low evaluations will be raised.

Randi Weingarten, president of the American Federation of Teachers, the national union to which the Washington local belongs, said in a statement that it is “very troubling when the district continues to reduce everything about students, educators and schools to a nameless, faceless algorithm and test score.”

Here’s the letter from Davis to Henderson, and following that is a statement from Weingarten as well as an IMPACT chart:

Dear Chancellor Henderson,
In an email message from Jason Kamras on Friday, December 20, 2013, I was informed that Mathematica Policy Research, a DCPS external partner, recently found a technical error that affected some teachers’ 2012-2013 Individual Value-Added (IVA) and Teaching & Learning Framework (TLF) evaluation scores. Needless to say, I was deeply disturbed by this preliminary information as well as the time at which it was provided—the day before the winter break.
In our continuing effort to be open, transparent and in accordance with our current collective bargaining agreement, the Washington Teachers’ Union (WTU) requests all information regarding the miscalculation of teachers’ SY-2012-‘13 Individual Value-Added (IVA) scores and changes made to teacher’s (IMPACT) evaluation scores for that time period. In addition, the WTU requests all information regarding errors made in any other year during the implementation of the IMPACT evaluation system.
Our collective bargaining agreement (CBA) allows the WTU to receive all information relevant to the enforcement of the agreement (10.1.1). Therefore, the WTU immediately requests the following information from DCPS regarding the miscalculation of teachers 2012-13 IVA and TLF IMPACT scores;
• a list of all teachers affected by these errors,
• copies of all correspondence sent to teachers regarding the miscalculation of their scores,
• the impact of the miscalculation on each teacher,
• the SY-2012-13 evaluation score of every DCPS teacher,
• the list of teachers who submitted challenges to their SY-2012-13 evaluations
• the list of all evaluations that were changed as a result of an evaluation challenge
• the list of teachers whose challenges were denied
• the DCPS letters of response to each teacher whose challenge was changed or denied
• a full description of the Mathematica error, the cause of the error, how and when the error was brought to the attention of DCPS officials, the number of teacher scores that were affected by the error,
• a copy of all DCPS communications sent to affected teachers
• each teacher’s SY2012-13 final evaluation: the initial evaluation that was sent to teachers and any changes to those evaluations that were subsequently made and documents explaining the reasons for the changes.
• a copy of the DCPS database reports showing teachers’ name, school, area of certification, actual assignment, IMPACT category, component scores (Teach, CSC, etc.).For teachers whose scores are based in part on value-added calculations, provide the formula and how each teacher’s score was generated
• a listing of all errors in any aspect of IMPACT score calculations in the 2012-13 IMPACT score calculations, and the weight of those errors in IMPACT points, including the notification of the errors and the DCPS response to those notifications.
• a list of each teacher’s final IMPACT score for 2012-13 and the new score that results from correction of the “error.” This includes teachers whose score changes would not move them into a new evaluation category, e.g. from “effective” to “highly effective.”
• all errors and the weight of those errors in IMPACT points that were previously discovered since the implementation of IMPACT and how those errors were corrected.
•  the names, addresses, email addresses and school assignments of teachers whose scores in 2012-13 scores would have been lowered if changes resulting from the correction of the error had were implemented, thereby moving the teacher into a lower evaluation category.


Please provide answers to the following questions:
1. to what extent were specific teachers affected, i.e., loss of pay, loss of step in crease or bonus, separation from duty, change in IMPACT category, etc.?
2. what are the specific groups of teachers affected by these or other errors? Elementary? Teachers with Individual student test scores? Teachers in a specific Ward of D.C.?
3. What are the specifics of the proposed remedies for each affected teacher?
4. What is DCPS’s plan for remedying the impact of the error on each affected teacher?
5. When will these remedies be instituted?
6. Was the error the result of actions taken by one or more employees of DCPS? If so, was the person or persons responsible for evaluating or reviewing teacher’s evaluations?
7. Were personnel actions taken against any administrator or private contractor due to these errors?
8. Have affected teachers been notified? If so, how and when?
9. What are the future communications plans regarding these errors?


Finally, the WTU requests copies of all communications between the school district and all third-party contractors regarding these errors and the IMPACT system. Along with general communications, this should include all technical and general reports related to IMPACT, the errors and teaching quality in DCPS. Thank you for your anticipated cooperation.


Elizabeth A. Davis
President, Washington Teachers’ Union


Statement from Randin Weingarten, president of the American Federation of Teachers:

We believe in D.C. public schools, and have worked with our local union and this mayor in many constructive ways, including on its very successful pre-K program. But there’s something very troubling when the district continues to reduce everything about students, educators and schools to a nameless, faceless algorithm and test score. This was clear in the Rhee era and led to widespread allegations of cheating. And now we see it with the troubling news that teachers’ evaluation scores were miscalculated—with a tremendous impact on the employment and wages of teachers and on our schools and students.
You can’t simply take a bunch of data, apply an algorithm, and use whatever pops out of a black box to judge teachers, students and our schools. And now, we have the disclosure that even the number was miscalculated, affecting dozens, if not hundreds, of educators. Our children deserve better.