The Washington Post

A shocking statistic about the quality of education research


In 1968, an article in Kappan magazine titled “The Need for Replication in Education Research” by education Professor Robert H. Bauernfeind said:

The principle of replication is the cornerstone of scientific inquiry. This principle holds that under similar conditions, one should obtain similar results. Replication has long been an essential aspect of research in the natural sciences, where science findings are not published until their repeatability has been demonstrated. In the natural science, the investigator may repeat his experiment 10 or 20 times, cross-comparing all results, prior to publishing his “findings”….

Yet the process of replication is much more vital in our field than in the natural sciences or even the biological sciences. The reason, simply, is that more things can go wrong in a behavioral research project than a physical research project. There is a higher probability that the findings of a single behavioral study might be in serious error, or might not be generalizable beyond the specific circumstances of the specific study…

That is why the findings of a new research study about research studies are so important — and cautionary.

Veteran teacher Larry Ferlazzo alerted me to Facts Are More Important Than Novelty: Replication in the Education Sciences,” published in the peer-reviewed Educational Research, a journal of the American Educational Research Association, and written by Matthew C. Makel of Duke University and Jonathan A. Plucker of the University of Connecticut.

Their study analyzed the “complete publication history of the current top 100 education journals ranked by 5-year impact factor.” The impact factor of an academic journal speaks to the average number of times an individual article has been cited in the Journal Citation Report in a year; a five-year impact factor would mean the average number of times pieces from a specific journal published in the previous five years were cited in the JCR. Journals with a higher number of citations are considered to be more important in their field than those with fewer citations.Their study says that despite the fact that replication of education research findings is vital to policymakers and educators, only 0.13 percent of education articles were replicated.

That’s 0.13 percent. Less than one percent. Less than half of one percent.

There’s more: While nearly 68 percent of the replicated studies were successful in reaching the same conclusions as the original studies, it was also true that replications “were significantly less likely to be successful when there was no overlap in authorship between the original and replicating articles,” according to an abstract of the report. When an entirely new team of researchers did a replication, they were successful in duplicating the results of the original 54 percent of the time. “The results,” the abstract said, “emphasize the importance of third-party, direct replications in helping education research improve its ability to shape education policy and practice.”

For more than a decade, school reformers have said that education policy should be driven by “research” and “data,” but there’s a big question about how much faith anyone should have in a great deal of education research. This is so not only because the samples are too small or because some research projects are funded by specific companies looking for specific results, but because in nearly all cases, it appears that nobody can be certain their results are completely accurate.

It is true that replication studies that confirm the original findings are not necessarily definitive and cannot resolve all issues about rigor, reliability, precision and validity of educational research. The report cites eight reasons that explain the lack of replication in education research, including the notion that “novelty equals creativity.” But it goes on to say this:

However, implicitly or explicitly dismissing replication indicates a value of novelty over truth… and  a serious misunderstanding of both science and creativity. If education research is to be relied upon to develop sound policy and practice, then conducting replications on important findings is essential to moving toward a more reliable and trustworthy understanding of educational environments.

Inside Higher Ed quotes Plucker in this excerpt from its story on the Mackel-Plucker report:

Neighboring fields in the social sciences – psychology, sociology, criminology – also suffer from a dearth of replications. But whereas psychology has weathered a number of fraud cases, the world of education research has had not a single fraud accusation in years, Plucker said. That’s a remarkable statistic, considering that conservative estimates place the number of educational researchers at 50,000 in the U.S. and 100,000 worldwide, according to the American Educational Research Association.

Education research’s spotless record is no accident, he said. It’s the result of scholars who aren’t checking each other’s work.

“I would love to believe that every single person doing education research around the world has ethics that are as pure as the driven snow,” Plucker said. “[But] the law of averages tells us there’s something out there.”

Valerie Strauss covers education and runs The Answer Sheet blog.



Success! Check your inbox for details. You might also like:

Please enter a valid email address

See all newsletters

Show Comments
Most Read



Success! Check your inbox for details.

See all newsletters

Your Three. Videos curated for you.
Play Videos
How to make Sean Brock's 'Heritage' cornbread
New limbs for Pakistani soldiers
The signature dish of Charleston, S.C.
Play Videos
Why seasonal allergies make you miserable
John Lewis, 'Marv the Barb' and the politics of barber shops
What you need to know about filming the police
Play Videos
The Post taste tests Pizza Hut's new hot dog pizza
5 tips for using your thermostat
Michael Bolton's cinematic serenade to Detroit
Play Videos
Full disclosure: 3 bedrooms, 2 baths, 1 ghoul
Pandas, from birth to milk to mom
The signature drink of New Orleans
Next Story
Valerie Strauss · August 20, 2014

To keep reading, please enter your email address.

You’ll also receive from The Washington Post:
  • A free 6-week digital subscription
  • Our daily newsletter in your inbox

Please enter a valid email address

I have read and agree to the Terms of Service and Privacy Policy.

Please indicate agreement.

Thank you.

Check your inbox. We’ve sent an email explaining how to set up an account and activate your free digital subscription.