(Correction: Changing attribution of paragraph about VAM, which was incorrectly attributed to the American Statistical Association report in an earlier version. )
There are three possibilities why New York Gov. Andrew Cuomo would ignore the position of the American Statistical Association on how not to use statistics to evaluate teachers.
One is that he doesn’t know the organization’s position. Two is that he knows but doesn’t understand it. Three is that he knows and understands but thinks he knows better.
Any of those would explain why the governor (and the Obama administration and other school “reformers”) would ignore warnings from the association and other assessment experts who say that using student test scores to evaluate teachers is not a valid assessment tool, and proceed to ratchet up the percentage by which student standardized test scores would be used to judge individual teachers.
In his recent State of the State Address, Cuomo advanced some major reforms to the state’s school reforms that were at odds with researchers, teachers and others over how to evaluate educators. (His embrace of school vouchers was also at odds with the position of the Obama administration and most other Democrats.) Cuomo said he wants to:
*Boost overall school funding by nearly 5 percent — but only provide the full increase if state legislators do what he wants on school reforms.
*Require that student standardized test scores account for a full 50 percent of a teacher’s evaluation rather than the current 20 percent. Because the current evaluation system, which was itself poorly implemented a few years ago, determined that very few teachers are “ineffective,” Cuomo apparently assumes that the problem with the system is that test scores aren’t a big enough factor.
–It is worth noting that other states have linked test scores to educator evaluation in recent years, a step many took to win federal Race to the Top funding as well a waiver from the Obama administration from the most onerous parts of No Child Left Behind. But many have been attempting to reduce the impact of test scores on evaluation rather than increase it, like Cuomo.
*Require that the other 50 percent of a teacher’s evaluation to come from observations made by school officials and outside educators brought in for the task.
— Assessment experts say that the best evaluation systems use multiple measures, and, in fact, some top-rated school systems in the country have had success with such systems that put no weight at all on student standardized test scores.
*Deny an “effective” rating to any teacher who is rated “ineffective” based on test scores, and award bonuses of up to $20,000 to teachers who are “highly effective” (thanks to test scores of their students).
Here’s what the American Statistical Association said last year in a statement and report (which you can read below) about the high-stakes “value-added method” (VAM) of evaluating teachers. It uses student standardized test scores to determine the “value” of a teacher through complicated mathematical formulas that purport to be able to isolate every factor that goes into student academic performance, including hunger and sickness:
Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores, and that the majority of opportunities for quality improvement are found in the system-level conditions. Ranking teachers by their VAM scores can have unintended consequences that reduce quality.
Education historian and advocate Diane Ravitch wrote about those unintended consequences:
“There will surely be unintended consequences, such as a diminishment in the number of people willing to become teachers in an environment where ‘quality’ is so crudely measured. There will assuredly be more teaching to the test.”
Furthermore, a joint statement by the American Educational Research Association and the National Academy of Education on “Getting Teacher Evaluation Right” says:
“With respect to value-added measures of student achievement tied to individual teachers, current research suggests that high-stakes, individual-level decisions, or comparisons across highly dissimilar schools or student populations, should be avoided. Valid interpretations require aggregate-level data and should ensure that background factors – including overall classroom composition – are as similar as possible across groups being compared. In general, such measures should be used only in a low-stakes fashion when they are part of an integrated analysis of what the teacher is doing and who is being taught.”
Teachers are pushing back on Cuomo’s evaluation proposals. A group called the Concerned Teachers of New York State has written an open letter that teachers are signing in droves. On its Web site, the group says it consists of teachers “who are deeply concerned that the policies that are being pursued by Governor Andrew Cuomo will be actively destructive to the public school system of New York State” and that they “do not represent the positions of any formal organization, anymore than we represent the positions of signatory teachers on any issues other than the ones explicitly mentioned in the Teacher’s Letter.”
Here’s the text of the open letter from the Concerned Teachers of New York State.
In your recent “State of the State” Address, you indicated that you want to pay an annual, $20,000 merit-pay-style bonus to teachers that are rated “Highly Effective” under your revised annual professional performance review (APPR) plan. You also proposed increasing the emphasis of state-derived exam-score-based metrics in a teacher’s APPR rating to 50%, and additionally developing scoring guidelines wherein no teacher can receive a rating greater than “Developing” if the exam-score-based rating is “Developing” or lower. We feel that these two proposals, if enacted, will be extremely damaging to the long-term interests of the public school system and students of New York State.
The increased emphasis your proposed plan puts on state exam scores is problematic for several reasons:
- We are hard-pressed to find any reference literature that supports staking such a large portion of a teacher’s overall evaluation rating on how his/her students perform on a standardized exam. We can, in fact, find no reasoned basis for the increased emphasis that you have proposed. It is therefore extremely unwise to tie continued employment and merit-pay-style incentives to such a questionable metric.
- The increased emphasis that your proposed rating system places on exam scores is particularly concerning when considered along with the unclear methodology by which such state-provided scores are generated. To this point, we are not aware of any public school teacher in New York State who is able to explain the methodology that the state’s Department of Education uses to determine these scores, much less why these scores are methodologically valid for determining so much of a teacher’s effectiveness rating. We are, however, very aware of the warnings that have been issued by the relevant professional associations of statisticians about the misuse, and overemphasis of exam-score-based ratings when determining a teacher’s effectiveness.
- We are similarly at a loss for any reference that supports your proposed scoring dynamic wherein it will not be possible for a teacher to receive an evaluation rating higher than “Developing” if the exam-score portion of the evaluation is not higher than “Developing”. This mode of scoring a teacher’s efficacy seems to eliminate the significance of classroom observations by trained administrative personnel from the equation. We are particularly concerned that the canonization of exam-score-derived ratings will lead to unintended incentivizing of bad pedagogical practices (e.g. “teaching to the test”).
- From a collegial standpoint, we worry about the effect of incentivizing student exam performance on the quality of the culture of teaching in this state. We firmly believe that the public school teacher corps of New York State is among the finest in the nation. We are not aware of any situation where exam-score-centered, merit-pay-style compensation has led to increased collegiality, sharing of resources, or willingness for teachers to take on the necessary work of educating the most traditionally underserved student populations in our nation’s public schools. We are aware of many situations where the implementation of merit-pay-style compensation has contributed to decreased collegiality, and engendered comparative resentment among members of the teacher corps. To our minds, the lack of merit-pay-style compensation in the New York State public school system is a tremendous virtue, and contributes to a willingness among teachers to work with those populations of students who are most challenged in their educations.
- We are concerned that the environment that is created by the increased emphasis of high-stakes testing on teacher-effectiveness ratings encourages school staff to act dishonestly with regard to the security of exam administration. The increased emphasis of high-stakes testing on teacher-effectiveness ratings has been repeatedly demonstrated to result in cheating scandals like those seen in Atlanta, Georgia, and Washington, D.C..
- Finally, we think that the structure of the system indicates that you are of the opinion that a vanishingly slim minority of the state’s ~200,000 educators are going to receive “Highly Effective” ratings under your proposed system. A back-of-the-envelope calculation demonstrates that if just 5% of the teacher corps of New York State (10,000 teachers) were to qualify as “Highly Effective”, the state would pay out $200,000,000 in compensation. Being that we do not see any indication of any amount near that number in your proposed budget, we must conclude that you believe that your system will designate a substantially lower number of teachers as “Highly Effective”. To that end, we are disappointed that you hold the public school educators of your state in such low regard. Then again, since you seem to be under the belief that we are motivated in our careers by money alone, we can not say that we are surprised.
These are the major concerns that we have with your proposed evaluation system/merit-pay scheme. While we are sure that you think you are doing us a favor, we are writing to let you know that we are not interested in your “bonus”. We suggest that you take the money that you have encumbered for this proposal and put it in to programs that will improve the quality of education throughout the state, without regard to teacher-effectiveness ratings. In the spirit of offering constructive alternatives, here are several that we urge you to consder:
- Implement a statewide grant program for teachers, so that teachers who are currently forced to turn to organizations like DonorsChoose.org have an avenue for project funding within the state Department of Education.
- Expand programs that reward the best work being done by teachers, and that evaluate that work holistically, rather than on the sole basis of state-exam scores. Legislate salary increases for teachers who complete rigorous professional certifications (e.g. National Board Certification), and expand the New York State Master Teacher Program to cover teachers of all grade-levels, and all subjects, rather than its current coverage of only STEM teachers in grades 6-12.
- Consider putting the money back into the state’s school aid formula to help cover some of the costs of the various unfunded mandates that have been implemented during your tenure, and the limited ability of districts to respond to the increased costs of offering the same level of services as they have traditionally done under the limiting influence of the tax-cap.
- Encourage teachers to work with the most traditionally underserved student populations in New York by removing any aspects of the APPR framework that may inadvertently punish teachers for making that choice with a diminished score. Additionally, work to make the quality of teaching positions that most directly serve these student populations as competitively attractive as the positions in New York’s wealthiest areas.
Most importantly, talk with us not at us. At current, we do not feel that you hear us in our concerns, or that you listen to our needs. We are very interested in having a Governor that we can talk to about how best to educate the students of New York. But we are not interested in being paid extra to watch as you degrade the quality of public education for the students of New York State. Please support us in our profession, and abandon these destructive proposals.
Very truly yours,
The Concerned Teachers of New York State
Here’s the American Statistical Association’s statement on using VAM to evaluate teachers: