Their study analyzed the “complete publication history of the current top 100 education journals ranked by 5-year impact factor.” The impact factor of an academic journal speaks to the average number of times an individual article has been cited in the Journal Citation Report in a year; a five-year impact factor would mean the average number of times pieces from a specific journal published in the previous five years were cited in the JCR. Journals with a higher number of citations are considered to be more important in their field than those with fewer citations.Their study says that despite the fact that replication of education research findings is vital to policymakers and educators, only 0.13 percent of education articles were replicated.
That’s 0.13 percent. Less than one percent. Less than half of one percent.
There’s more: While nearly 68 percent of the replicated studies were successful in reaching the same conclusions as the original studies, it was also true that replications “were significantly less likely to be successful when there was no overlap in authorship between the original and replicating articles,” according to an abstract of the report. When an entirely new team of researchers did a replication, they were successful in duplicating the results of the original 54 percent of the time. “The results,” the abstract said, “emphasize the importance of third-party, direct replications in helping education research improve its ability to shape education policy and practice.”
For more than a decade, school reformers have said that education policy should be driven by “research” and “data,” but there’s a big question about how much faith anyone should have in a great deal of education research. This is so not only because the samples are too small or because some research projects are funded by specific companies looking for specific results, but because in nearly all cases, it appears that nobody can be certain their results are completely accurate.
It is true that replication studies that confirm the original findings are not necessarily definitive and cannot resolve all issues about rigor, reliability, precision and validity of educational research. The report cites eight reasons that explain the lack of replication in education research, including the notion that “novelty equals creativity.” But it goes on to say this:
However, implicitly or explicitly dismissing replication indicates a value of novelty over truth… and a serious misunderstanding of both science and creativity. If education research is to be relied upon to develop sound policy and practice, then conducting replications on important findings is essential to moving toward a more reliable and trustworthy understanding of educational environments.
Inside Higher Ed quotes Plucker in this excerpt from its story on the Mackel-Plucker report:
Neighboring fields in the social sciences – psychology, sociology, criminology – also suffer from a dearth of replications. But whereas psychology has weathered a number of fraud cases, the world of education research has had not a single fraud accusation in years, Plucker said. That’s a remarkable statistic, considering that conservative estimates place the number of educational researchers at 50,000 in the U.S. and 100,000 worldwide, according to the American Educational Research Association.Education research’s spotless record is no accident, he said. It’s the result of scholars who aren’t checking each other’s work.“I would love to believe that every single person doing education research around the world has ethics that are as pure as the driven snow,” Plucker said. “[But] the law of averages tells us there’s something out there.”