U.S. math literacy on the international PISA exam in 2015 was below the international average.

For years now, some economists and policymakers have tried to make the case that there is an important link between student standardized test scores and economic growth in the United States. They do this with complex mathematical formulas that most people can’t even pretend to follow.

Critics have argued repeatedly that the link doesn’t really exist, but the debate continues, especially when new scores from the Program for International Student Assessment, or PISA, are released. PISA is given by the Organization for Economic Cooperation and Development every three years to 15-year-olds around the world in reading, math and science. Whenever the scores are released — the last time was in late 2016 —  U.S. students inevitably come out nowhere near the top. Americans have never done well on international tests, whatever the state of the economy, but that gets lost in the “sky-is-falling” reactions to scores.

One of the leaders of the movement linking student test scores to the country’s economic health is Eric Hanushek, who is also a pioneer in creating accountability systems that evaluate teachers by their students’ test scores. (For those who like rankings, he is No. 11 on Rick Hess’s 2017 Edu-Scholar Public Influence Rankings, which list 200 university-based scholars in this country deemed by Hess’s methodology to be the most influential in educational policy and practice.)

Hanushek first concentrated on domestic data sets, but then began looking at the issue through an international lens. For example, in a 2015 Foreign Affairs article that he co-wrote, Hanushek took on the United Nations, arguing that a package of development goals set by the organization was short on substance and that its approach to education was “insufficient.” Part of his analysis involved PISA scores, saying:

In a recent study of cognitive skills and inclusive development, we used data from 76 middle- and high-income countries to estimate the economic implications of raising the level of each country’s unskilled population to Level 1 in mathematics and science on the OECD’s Program for International Student Assessment (PISA). This level represents modern functional literacy and is a useful measure of the basic skills needed for economic participation.

Hanushek and others use this international “evidence” — test scores of teenagers — to argue that the United States is missing out on economic growth and that a revamping of the public education system is necessary for the country’s economic health.

Below is a new effort to look at what is real — and not real — about the link between international test scores and the country’s economic health. It specifically cites Hanushek’s work.  It was written by Jeremy Rappleye and Hikaru Komatsu, both of whom are on the faculty  of the Graduate School of Education at Kyoto University in Japan. Their most recent work, from which this is adapted, is an article titled “A New Global Policy Regime Founded on Invalid Statistics? Hanushek, Woessmann, PISA, and economic growth (2017)” in the Comparative Education journal.

Incidentally, I recently published this related post, titled, “What the numbers really tell us about America’s public schools.” It was written by David Berliner, an educational psychologist who is a former dean of the school of education at Arizona State University and a past president of both the American Educational Research Association and the Division of Educational Psychology of the American Psychological Association. Berliner wrote:

For many years I have been writing about the lies told about the poor performance of our students and the failure of our schools and teachers. Journalists and politicians are often our nations’ most irritating commentators about the state of American education because they have access to the same facts that I have.

They all can easily learn that the international tests (e. g. PISA, TIMSS, PIRLS), the national tests (e. g. NAEP), the college entrance tests (e. g. SAT, ACT), and each of the individual state tests follow an identical pattern. It is this: As income increases per family from our poorest families (under the 25th percentile in wealth), to working class (26th-50th percentile in family wealth), to middle class (51st to 75th percentile in family wealth), to wealthy (the highest quartile in family wealth), mean scores go up quite substantially.

In every standardized achievement test whose scores we use to judge the quality of the education received by our children, family income strongly and significantly influences the mean scores obtained.

Here’s the new piece by Rappleye and Komatsu:

 

By Jeremy Rappleye and Hikaru Komatsu

Does the performance of U.S. students on international assessments of math and science matter? Do mediocre rankings foreshadow future American economic decline? Sound familiar?

Since the 1950s, questions about the comparative performance of American pupils have been a perennial feature of American education debates.  After learning from European powers for more than a century (think Horace Mann’s push in the 1840s for the Prussian model), America’s postwar prosperity and preeminence generated a new confidence.  The United States should and would lead the world.

But such global expectations also carried consequences: It made Americans excessively anxious, perhaps even neurotically vigilant for would-be contenders.

The first comparison came with the skilled Soviets. When Sputnik went into orbit in the fall of 1957, then President Eisenhower responded less than a year later with the National Defense Education Act. It sought to increase secondary school enrollments and bolster research in the areas of science, technology, and engineering. Soon the wider public learned  “What Ivan Knows that Johnny Doesn’t” (1961). “By the time American schoolchildren get Jack and Jill up that hill,” author Arthur Trace admonished, “Soviet children of the same age will probably be discussing the hill’s attitude, mineral deposits and geo-political role in world affairs.” Such comparisons set the tempo for future reform cycles.

The second time that the postwar ground bass of global competition became distinctly audible was the 1980s.  With the 1983 release of “A Nation at Risk,” Americans again heard dire warnings that their education system would lead to national decline:

“Our nation is at risk. Our once unchallenged preeminence in commerce, industry, science, and technological innovation is being overtaken by competitors throughout the world.”

Gone were the Soviets, however, replaced by the efficient, disciplined, and annoyingly polite Japanese.  Persuaded by then President Reagan’s rhetoric and apparently confirmed by results from the first large-scale international test (the Second International Mathematics and Science Study, 1982), Americans again became anxious that its public schools were costing the country its economic preeminence.  Thus in 1989, then President George H.W. Bush and like-minded governors (including then Gov. Bill Clinton) promised to make American pupils first in international tests by the year 2000.

Today — some three decades onward — the same basso ostinato is becoming strongly audible for the third time. The OECD’s Program for International Student Achievement (PISA) has commanded progressively greater attention in America’s domestic education debates since the first results in 2000.  Recent discussion of the “Finnish model” and Singapore Math trace their origins to the triennial PISA rankings, as do recent popular accounts of education such as “The Smartest Kids in the World — And How They Got That Way” (Ripley, 2013) and “Cleverlands: The Secrets Behind the World’s Education Superpowers” (Crehan, 2016).

Unsurprisingly, some are now connecting the new PISA rankings to the long-standing worries about competition and national decline. Representative here is the Brookings Institution’s “Endangering Prosperity: A Global View of the American School” (Hanushek, Peterson, Woessmann 2013). It warns that the lackluster performance by American students on PISA tests will soon weigh heavily on America’s economic future. To avert this, the report argues sweeping structural reforms are now urgent.  Gone are the Soviets and Japanese, this time replaced with faceless OECD numbers: a rich data set of OECD countries (roughly 60), longitudinal test scores, and GDP growth rates worldwide since the 1960s.

But this new rendition, while well worn, is decidedly more persuasive than previous comparisons. Why? The certainty of statistical correlations stands in for the ambiguity of what can be observed in foreign classrooms. Specifically, “Endangering Prosperity” comes out of a vast body of persuasive statistical work conducted by Eric Hanushek (Stanford University) and Ludger Woessman (University of Munich) that began in the late 1990s. Their project analyzes the relationship between previous international test results and current PISA test scores, then connects it all with economic growth worldwide from 1960 to 2000. Distilled down into the one key finding:

“With respect to magnitude, one standard deviation in test scores (measured at the OECD student level) is associated with an average annual growth rate in GDP per capita two percentage points higher over the forty years that we observe” (The Knowledge Capital of Nations by Hanushek and Woessman 2015, p.44).

Their key point is that the relationship between PISA-style test scores and GDP over the last 50 years is “causal” (p.65).

In plain English for jittery American policymakers: International evidence shows that successfully raising tests scores will produce significantly higher future GDP growth. But those who are “out learned” will be overtaken.

How much growth? These projections suggest quite a bit. In the case of a country currently performing at the OECD mean that successfully raised its test scores to the level of top PISA performers through 20 years of aggressive reforms, its GDP would be 5 percent greater.

In other words, if American pupils (PISA 2000) could crack the level of Finnish or Korean schoolchildren (PISA 2006) it could cash in on a 5 percent larger GDP.  Again in plain English: If the United States had just stayed the course set by George H.W. Bush and governors in 1989 and successfully achieved the promised 50-point gain, Hanushek and colleagues argue, U.S. GDP would already be more than 5 percent greater (Figure 1).


Figure 1. Economic benefits of reform (Education and Economic Growth (2008), figure redrawn by authors)

 

This is highly significant: That much GDP growth could have paid for all primary and secondary school spending and put more money in everyone’s pocket. Gains would be equivalent to “20 percent higher paychecks for the average worker over the entire 21st century,” calculates Hanushek, before arguing that “not only could this solve our current fiscal and distributional woes, but it also could establish our future economic and international leadership.” (Why the US Results on PISA Matter, 2014)

Today we can look back to the collapse of the Soviet Union and the bursting of the Japanese asset bubble to argue that international comparisons of student achievement are poor predictors of economic futures. Both of these nations were once used to stoke fear among American parents, and yet that fear was never justified.

But what are we to make then about causal claims based on numbers and longitudinal data?  While single competitors might fall, have we actually been blind to more general worldwide patterns that have remained robust? Maybe the postwar pitch was on to something after all?

Anxious about all this, we conducted our own study that looked more closely at the causality claims. We utilized the exact same sample of countries, data, and methods utilized by Hanushek and Woessmann. The only adjustment we made was selecting a more reasonable time period for the economic growth calculation.

Once when we did this, however, we came up with a disturbing result: The relationship becomes so dramatically weak it cannot support the claims.


 

Figure 2. Relationships between test scores and economic growth: (a) our replication of Hanushek’s findings and (b) our findings.

The key point is time mismatch. And one need not be an educational researcher or statistics geek to understand the logic. Hanushek compared test scores and economic growth over the same time period (1960-2000), reporting the strong relationship he found (the left side of Figure 2). However, it logically takes at least a few decades for students to occupy a major portion of the workforce. So we compared test scores for one period with economic growth in a subsequent period (1995-2014). Surprisingly, the relationship, which was once looked so strong, now looked suspect (the right side of Figure 2).

This finding not only refutes a tight test score-GDP growth link, but also unmasks causality as simply statistical coincidence. And coincidence can certainly not support the notion that aggressively raising test scores will eventually generate enough extra money to “solve our current fiscal and distributional woes.” Nor can it support the claim that helping American children ace PISA-style tests will lead to “20 percent higher paychecks” for every American this century — although we certainly wish it did.

So when we ask: Does the mediocre rankings of U.S. pupils on international learning assessments foreshadow future American economic decline? We answer: The evidence does not support these claims, even when we use the same data and methods.

As such, contemporary anxieties derive less from evidence, more from the stubborn beat of global competition that has, unfortunately, come to dominate American educational culture. Research “findings” such as these reveal more about cultural predilections than reality itself. This becomes most clear when we viewed against the larger historical backdrop sketched here. What might actually endanger prosperity then is hastily implementing structural reforms based on truncated comparisons and flawed statistical analyses.

To be clear, the United States should keep trying to learn about educational policy and practice elsewhere. But international comparisons should be understood as means of learning about ourselves, not as a baton to bring domestic debates to an anxious crescendo.

In doing so, we might discover that nowhere are schools viewed as a means to economic superiority as strongly as in the United States. By replacing our competitive mind-set with a learning one, perhaps we might break from the cultural cadence of global expectations pacing postwar America education.