wpostServer: http://css.washingtonpost.com/wpost

The Post Most: Local

Answer Sheet
Posted at 12:20 PM ET, 12/13/2011

Teach for America ‘research’ questioned

This was written by Philip Kovacs, assistant professor of education at the University of Alabama, Huntsville assistant professor Philip Kovacs. A version of this was first published on Anthony Cody’s blog, Living in Dialogue, on the Education Week Teacher site.

He starts the post by referring to Teach for America’s training system, which involves taking newly minted college graduates and putting them through five weeks of intensive summer training before sending them into classrooms in some of the country’s most troubled schools.

Kovacs has tried to get the Hunstville school board to re-examine its decision to spend $1.7 million to bring Teach For America teachers to its public schools after it laid off 300 teachers over the past two years.

By Philip Kovacs

Recently I exchanged emails with a Teach for America employee in my city. On my last exchange, I tried to press her to answer at least one of my questions.

"Given the choice, would you see a doctor with 5 weeks of training or a certified doctor? A lawyer? An actuary?"

Answering with a ‘yes’ would be absurd. Answering with a ‘no’ would indicate a blatant disrespect for teachers.

Unfortunately this disrespect is exactly what we have going on in our country at this time: a blame-the-teacher mentality that ignores real world issues and concerns.

The TFA employee directed me to the organization's "research" page where TFA claims this: "A large and growing body of independent research shows that Teach For America corps members make as much of an impact on student achievement as veteran teachers."

This claim, based on the "studies" supplied by TFA, is misleading at best and demonstrably false at worst. I read all of the 12 "studies" available on TFA's website, and here is what I found.

Four of the 12 "studies" are irrelevant to the argument re: "make as much of an impact on student achievement as veteran teachers." Of these four:

One, “Creating a Corps of Change Agents ,” is a fluff piece from Education Next that discusses the high rate of entrepreneurs who come from TFA;

The second is a peer-reviewed piece, “The Price of Misassignment: The Role of Teaching Assignments in Teach For America Teachers' Exit from Low Income Schools and the Teaching Profession,” which discusses improving TFA retention ;

The third, “ Teacher Characteristics and Student Achievement:
Evidence from Teach For America,” discusses predicting outcomes at the time of TFA hire. (This one could have gone under "problematic" as the front page contains the disclaimer "PRELIMINARY AND INCOMPLETE" in all caps.)

The fourth is another peer-reviewed article, “ Assessing the Effects of Voluntary Youth Service: The Case of Teach for America” presents evidence against TFA's claim that TFAers go on to "pro social" jobs.

There are problems with seven of the 12 studies that include some some methodological flaws. In fact, two of the seven acknowledge such flaws and warn the reader against making judgments based on their data.

And some of the seven show mixed results, i.e. TFA recruits are better at math than some teachers in some cases but are not better in other subjects, or they are better than novice teachers but not better than those with experience, etc.

Importantly, all of the seven studies that show mixed or problematic results are based on the use of Value Added Measurement (VAM). Here is a link to one peer-reviewed research paper, “Teacher Effects and Teacher Effectiveness, a Validity Investigation of the Tennessee Value Added System ,” which argues that there are "several logical and empirical weaknesses of the system" used to evaluate teachers in Tennessee, the system which found its way to TFA's "research" page.

VAM is flawed at best, as argued in this report from the Annenberg Institute, an institute that can hardly be called partisan or pro-status quo, though some readers will no doubt level the criticism. Education historian Diane Ravitch, discussing the Annenberg report, asks an important question: "[Dr. Corcoran] describes a margin of error so large that a teacher at the 43rd percentile (average) might actually be at the 15th percentile (below average) or the 71st percentile (above average). What is the value of such a measure? Why should it be used at all?"

One of the issues educator Anthony Cody rightly addresses, however, is that the more we talk about VAM, the more we reify it as an accurate tool for determining teacher effectiveness, which it simply isn't.

A teacher raising student scores from the 15th to 25th percentile is going to look, to bean counters, much more effective than a teacher who raises student scores from the 85th to the 90th.

Which teacher is more effective? That's debatable, but it is the type of debate that happens when people go to football games and stare at the scoreboard for two hours. Both teachers might be equally effective, given the students that they are teaching and the conditions under which the students live and the teachers work. The teacher with the smaller gain might be more effective, but to really know, you'd have to know something about the teams and you would have to watch the game.
Finally, one TFA cited “study” is overwhelmingly positive towards the organization, but that "study" is actually a one-page summary from a survey of principals. The questions and data are not available.

It turns out that TFA left some reports off of its website. They aren't very flattering though, so I understand.

See, for example, “The Effectiveness of ‘Teach for America’ and Other Under-certified Teachers ,” by Ildiko Laczko-Kerr and David C. Berliner, and Does teacher preparation matter? Evidence about teacher certification, Teach for America, and teacher effectiveness ,” by Linda Darling-Hammond et al. Note that both of these research papers are from the Education Policy Analysis Archives, one of the two peer-reviewed journals included on TFA's website.

As to the importance — or non-importance — of peer review, scholars and scientists have the mechanism in place to make sure research is sound and people aren't simply making things up and convincing others that they have found the cure for cancer, created a miracle drug like Vioxx, cloned a sheep, or narrowed the achievement gap.

Two of the 12 studies on TFA's website are peer-reviewed. Both are, however, irrelevant to TFA's claim "that Teach For America corps members make as much of an impact on student achievement as veteran teachers."

What is troublesome here is that we now live in a world where foundations and organizations have millions of dollars to spend lobbying and at the same time can bypass peer review in order to make a case for whatever they are selling. If you have enough money, science no longer matters. For more on this ask the scientists trying to address global warming.

Here is what I can say with some certainty based on TFA's "reports:" In some cases, in some places, and in some grades, TFA might produce better results on math tests than traditionally certified, novice teachers.

The rest is very debatable. The "research" is certainly not worth Huntsville paying an extra $1.7 million for Teach for America recruits, and it is certainly not good enough for the children who need experienced teachers.

The data showing experience matters is overwhelming, something the Coalition for Teaching Quality recently brought to the attention of Sen. Bernie Sanders. Consider taking the time to watch the entire briefing. TFA does not come out so well.

But don't take the CTQ's word for it, as TFA acknowledges that experience matters on its "research" page. Check out the "Portal Report" (which is a pdf): "Teachers with 4 years or more experience out perform teachers with 1 year of experience on 9 out of 10 indicators."

-0-

Follow The Answer Sheet every day by bookmarking http://www.washingtonpost.com/blogs/answer-sheet. And for admissions advice, college news and links to campus papers, please check out our Higher Education page. Bookmark it!

By  |  12:20 PM ET, 12/13/2011

 
Read what others are saying
     

    © 2011 The Washington Post Company