It is always thrilling, if a little sad, when social scientists gather to solve the insoluble. Few of the scholarly conference reports I have read are as brave in the face of bad odds as the National Academy of Education’s recent 94-page publication, “International Education Assessments: Cautions, Conundrums, and Common Sense.”
Anyone interested in how research affects policy should read this document. At two D.C. workshops, experts on statistics and education brought together by the academy discussed the flawed ways we report on American school quality and how we can make that better.
The report highlighted this headline on an NPR Web report in 2013: “U.S. Students Slide in Global Ranking on Math, Reading, Science.” The story said: “American 15-year-olds continue to turn in flat results in a test that measures students’ proficiency in reading, math and science worldwide, failing to crack the global top 20.”
I can imagine the exasperated sighs that greeted that example. The academy report said: “The use of the word ‘slide’ in the headline gives the mistaken impression that U.S. scores are declining, although the article goes on to explain that the average scores remain relatively flat. . . . Articles like this one certainly capture readers’ attention, for better or for worse.”
The academy report focused on the worse. We in the media have heard this before, and not just on the education page. Our headlines are slanted. Our stories omit important facts. Context is sacrificed in favor of some alleged disaster that will capture online attention.
This does happen, although all the reporters I know have tried to make our work accurate. The workshop leaders from the nonprofit National Academy of Education, dedicated to improving educational research, recognized this. To my shock, they even included some first-rate journalists, something you don’t see academics do very often.
Nicholas Lemann, former dean of Columbia University’s journalism school, gave a detailed account of how falling advertising revenue and declining newsroom staffs had crippled education reporting. Kevin Carey, who directs the education policy program at the New America think tank, described his efforts to challenge the common view that no matter how bad our K-12 schools are, our universities lead the world.
Most interesting: The researchers confessed that the weakness of their work sometimes made it difficult to convince people that it reflected the truth. They used the word “causality” a lot. People want to know what causes U.S. students to learn less than children in Japan, Singapore and Finland. As I typed that last sentence, I reminded myself that we really don’t know how valid those comparisons are.
Here was the report’s key conclusion: “Most researchers believe causality cannot be firmly established without randomized controlled experiments or rigorous quasi-experiments, which are typically not realistic in educational settings at this scale.”
Exactly. It is not practical to recruit 30 sets of twins, teach Chinese to one in each pair, stick them in a Shanghai school, and then see how they do compared to their brothers and sisters at a school in New York. We have to do our best with the feeble tools we have.
There is also one overarching truth about the finest scholarly work that wasn’t covered in this report.
I wish the academy had consulted Timothy A. Hacsi, a historian at the University of Massachusetts Boston. Sixteen years ago in his landmark book “Children As Pawns: The Politics of Educational Reform,” he explained why the best research on issues such as class size and social promotion had little effect on how officeholders handle those issues.
“Politicians define the public good largely in the terms espoused by those who voted for them, by the stand of their political party, and by those who fund their campaigns,” he wrote. “Evidence about what works is usually well down the list of factors influencing policy.”
I hope the scholars will continue to enlighten us. We journalists will try to make their findings as clear and nuanced as we can. But the people who have the money and power to make changes are unlikely to pay much attention to us, something to keep in mind as we argue over how to make schools better.