The Washington PostDemocracy Dies in Darkness

Study: Test-score gains don’t mean cognitive gains

Placeholder while article actions load

In a finding that should give pause to backers of standardized test-based school reform, a new study by neuroscientists at three major universities shows that students who achieved  the highest gains on standardized tests did not show the same gains in the ability to analyze material and think logically.

The research, conducted by scientists from the Massachusetts Institute of Technology, Harvard University and Brown University and funded by the Bill and Melinda Gates Foundation and the National Institutes of Health, says that strategies that schools use to boost scores on the Massachusetts Comprehensive Assessment System exams did nothing to help students improve in the development of what is called “fluid intelligence” skills, or cognitive gains.

Modern school reform has been wrapped around the notion that “accountability” of students, schools and teachers can be achieved by monitoring the scores students receive on high-stakes multiple-choice standardized tests. Assessment experts have long said that most of the standardized tests that have been in use for many years do a poor job of evaluating the range of things students learn (and for that and other reasons, should not be used for high stakes decisions).

This study goes beyond those findings, seeming to “narrow down the possibilities of what educational interventions are achieving,” Daniel Willingham, a professor of psychology at the University of Virginia who was not part of the research team, was quoted as saying by a report on study published on the MIT website.

The researchers looked at data for nearly 1,400 eighth-grade students in Boston’s traditional public schools as well as charter schools and “exam schools” (which admit students based on grades and test scores)  with the goal of trying to find a link between test score gain and improvement in things such as critical thinking and memory retention, the MIT report said.

The website report says:

“Our original question was this: If you have a school that’s effectively helping kids from lower socioeconomic environments by moving up their scores and improving their chances to go to college, then are those changes accompanied by gains in additional cognitive skills?” says John Gabrieli, the Grover M. Hermann Professor of Health Sciences and Technology, professor of brain and cognitive sciences, and senior author of a forthcoming Psychological Science paper describing the findings…
…The researchers calculated how much of the variation in MCAS scores was due to the school that students attended. For MCAS scores in English, schools accounted for 24 percent of the variation, and they accounted for 34 percent of the math MCAS variation. However, the schools accounted for very little of the variation in fluid cognitive skills — less than 3 percent for all three skills combined.
In one example of a test of fluid reasoning, students were asked to choose which of six pictures completed the missing pieces of a puzzle — a task requiring integration of information such as shape, pattern, and orientation.
“It’s not always clear what dimensions you have to pay attention to get the problem correct. That’s why we call it fluid, because it’s the application of reasoning skills in novel contexts,” says Amy Finn, an MIT postdoc and lead author of the paper.
Even stronger evidence came from a comparison of about 200 students who had entered a lottery for admittance to a handful of Boston’s oversubscribed charter schools, many of which achieve strong improvement in MCAS scores. The researchers found that students who were randomly selected to attend high-performing charter schools did significantly better on the math MCAS than those who were not chosen, but there was no corresponding increase in fluid intelligence scores.


A paper detailing the findings of the study is being published in Psychological Science, a journal of the Association for Psychological Science.