The fetishization of international test scores

(Bloomberg)
(Bloomberg)

Not one, not two, but 10 national educational organizations are planning to host a blowout digital event to talk about (what else?) international standardized test scores. There’s even a new Web site just for PISA Day, called, you won’t be surprised to learn, PISADay.org.

The event is being held on Dec. 3, the same day as the release of the latest scores from PISA, the  Program  for International Student Assessment,  a test of reading, math and science given every three years to 15-year-old students in more than 65 countries and education systems by the Organization for Economic Cooperation and Development.

The new Web site says representatives from 10 organizations will analyze the results, as well as “their implications for U.S. education policy.”  They are:  Alliance for Excellent Education, Achieve, ACT, America Achieves, Asia Society, Business Roundtable, College Board, Council of Chief State School Officers, National Board for Professional Teaching Standards and National Center on Education and the Economy.

With so many organizations hosting, it makes you wonder how anybody is going to get in a word edgewise during the conversation.  Beyond that, the real question is why we keep placing so much importance on where U.S. students stand next to international students, given that there are real problems with making comparisons.

First of all, judging just about anything important on the sole basis of test scores is never a good idea. That’s not just me talking; assessment experts say it over and over and over.

Furthermore, U.S. students have never performed on international tests — including PISA. Public schools have long been blamed when something happens to challenge the country’s standing in the world. It happened when the Soviet Union launched the first space satellite, Sputnik, in 1957, and in 1983 when the Japanese automobile industry was booming and America’s wasn’t. Of course when the U.S. economy was roaring in the 1990s, you never heard anybody thank the public schools, but never mind.

There are also questions about whether the test scores show what they pretend to show. A report released early this year by Martin Carnoy of the Stanford University’s Graduate School of Education and Richard Rothstein of the Economic Policy Institute raised questions about whether the average scores  in the 2009 PISA were reported lower than they should have been.

The last time PISA results were released — 2009 scores made public in 2010 —  the United States wasn’t anywhere near the top, a performance that was pretty much like it has for years and years and years, when the U.S. economy was roaring and when it wasn’t.  The sky was falling, if you believed all of the cries of horror from policy makers and politicians.

Shanghai, participating for the first time in PISA, came out on top in all three areas out of about 65 countries and other education systems, and one education pundit actually compared the news to the attack on Pearl Harbor. If it sounds hysterical, that’s because it was.

What was ignored was that Shanghai doesn’t represent all of China and that the Chinese education system has been geared toward test-taking. And, most interestingly, Chinese education officials have recently announced a major school reform that is aimed at deemphasizing standardized test scores. Why? They believe it has stifled creativity and led to a narrow curriculum and bored students. Have you heard a single school reformer in the United States talk about that?

There’s another issue that gets overlooked in PISA score discussions: American students who took PISA in 2009 and attended low-poverty schools scored on average as high as the highest scorers in the world. American students in high-poverty schools scored far lower than the average.

William J. Mathis, managing director of the National Education Policy Center and a former Vermont superintendent, wrote in this 2010 post:

As OECD Paris-based official Michael Davidson said in National Public Radio comments, “One of the striking things is the impact of social background on (U.S.) success.”

Twenty percent of U.S. performance was attributed to social background, which is far higher than in other nations. Davidson went on to point out that the United States just does not distribute financial resources or quality teachers equally. In a related finding, students from single-parent homes score much lower in the United States than they do in other countries. The 23-point difference is almost a year’s lack of growth.

Education historian and activist Diane Ravitch wrote in her new book, “Reign of Error“:

Kevin Baker, who worked for many years as an analyst at the U.S. Department of Education, asked, “Are international tests worth anything?” Do they predict the future of a nation’s economy? He reviewed the evidence and concluded that for the United States and about a dozen of the world’s most advanced nations, “standings in the league tables of international tests are worthless. There is no association between test scores and national success, and, contrary to one of the major beliefs driving U.S. education policy for nearly half a century, international test scores are nothing to be concerned about. America’s schools are doing just find on the world scene.

Baker argued that the purveyors of doom and gloom were committing the “ecological correlation fallacy.” It is a fallacy to generalize that what is good for an individual (a higher test score, for example) must be right for the nation as a whole.  Maybe it is, maybe it isn’t, he said, but evidence, not just an assumption, is needed to make the case.”

His research found no reliable connection.

Finally, it’s always a good idea to look back at what the late, great social scientist Gerald Bracey wrote about international comparisons. Bracey was director of research, evaluation and testing for the Virginia Department of Education from 1977 to 1986, as well as a trained psychologist who was the leading critic of how today’s tests measure success. He authored numerous articles and books, including “Reading Educational Research: How to Avoid Getting Statistically Snookered.”

So the U.S. is not #1 in mathematics or science testing. So what?

So, very little.

First, comparing nations on average scores is a pretty silly idea. It’s like ranking runners based on average shoe size or evaluating the high school football team on the basis of how fast the average senior can run the 40-yard dash. Not much link to reality. What is likely much more important is how many high performers you have. On both TIMSS math and science, the U. S. has a much higher proportion of “advanced” scorers than the international median although the proportion is much smaller than in Asian nations.

This was not true on PISA, another international comparison that tests 15-year-olds. Only 1.5% of American students scored at the highest level compared to top performing New Zealand at 4% and second place Finland at 3.9%.

Yet the proportion of Americans at the highest level meant that 70,000 kids scored there compared to about 2,000 for New Zealand and Sweden. No one else even came close — Japan was second with about 33,000 top performers. These are the people who might end up creating leading edge technology in the future. Who cares if Singapore, with about the same population as the Washington Metro Area, and Hong Kong, with about twice that number, score high?

There aren’t many people there. (And, as journalist Fareed Zakariya found out, the Singapore kids fade as they become adults. More about that in a moment). The bad news is that the U. S., on PISA anyway, had many more students scoring at the lowest levels; these kids likely can’t compete for the good jobs in the country.

Second, test scores, at least average test scores, don’t seem to be related to anything important to a national economy. Japan’s kids have always done well, but the economy sank into the Pacific in 1990 and has never recovered.

 

To be fair, the Dec. 3 digital blow event being hosted by the 10 organizations is supposed to be “the start of an ongoing initiative that will look beyond international rankings to learn how to improve education in the United States,” according to Bob Wise, president of the Alliance for Excellent Education (as quoted in a release about the event).  The fact that this new conversation has to start on PISA test data release day tells you how important the results really are to school reformers.

So expect to hear more cries of disaster on Dec. 3, when the United States is nearly certain to be somewhere in the middle (or worse) on the latest PISA scores. You will hear reformers call for more reform — totally ignoring the fact that for well over a decade American school reform has been centered on standardized tests and other reforms that haven’t moved the  international test score needle.

Will it occur to them that their efforts aren’t working?

Here are some other posts on the subject that may interest you:

U.S. scores on international test lowered by sampling error: report

“We’re Number Umpteenth!”: The myth of lagging U.S. schools

China’s 10 new and surprising school reform rules

Hysteria over PISA misses the point

How Shanghai topped PISA rankings — and why it’s not big news in China

Do international test scores make sense?

Valerie Strauss covers education and runs The Answer Sheet blog.
SECTION: {section=local, subsection=null}!!!
INITIAL commentConfig: {includereply=true, canvas_permalink_id=washpost.com/8bvh5zpd9k, allow_comments=true, commentmaxlength=2000, includeshare=true, display_comments=true, canvas_permalink_app_instance=m6yzjj840m, display_more=true, moderationrequired=false, includefeaturenotification=true, defaultsort=reverseChronological, canvas_allcomments_id=washpost.com/km4ey0dajm, comments_period=14, includevoteofftopic=false, allow_videos=false, includesorts=true, markerdisplay=post_commenter:Post Commenter|staff:Post Writer|top_commenter:Post Forum|top_local:Washingtologist|top_sports:SuperFan|fact_checker:Fact Checker|post_recommended:Post Recommended|world_watcher:World Watcher|cultuer_connoisseur:Culture Connoisseur|weather_watcher:Capital Weather Watcher|post_contributor:Post Contributor, childrenitemsperpage=3, includeheader=true, includeverifiedcommenters=true, defaulttab=all, includerecommend=true, includereport=true, maxitemstop=2, source=washpost.com, allow_photos=false, maxitems=7, display_ugc_photos=false, includepause=true, canvas_allcomments_app_instance=6634zxcgfd, includepermalink=false}!!!

UGC FROM ARTICLE: !!!

FINAL commentConfig: {includereply=true, canvas_permalink_id=washpost.com/8bvh5zpd9k, allow_comments=true, commentmaxlength=2000, includeshare=true, display_comments=true, canvas_permalink_app_instance=m6yzjj840m, display_more=true, moderationrequired=false, includefeaturenotification=true, defaultsort=reverseChronological, canvas_allcomments_id=washpost.com/km4ey0dajm, comments_period=14, includevoteofftopic=false, allow_videos=false, includesorts=true, markerdisplay=post_commenter:Post Commenter|staff:Post Writer|top_commenter:Post Forum|top_local:Washingtologist|top_sports:SuperFan|fact_checker:Fact Checker|post_recommended:Post Recommended|world_watcher:World Watcher|cultuer_connoisseur:Culture Connoisseur|weather_watcher:Capital Weather Watcher|post_contributor:Post Contributor, childrenitemsperpage=3, includeheader=true, includeverifiedcommenters=true, defaulttab=all, includerecommend=true, includereport=true, maxitemstop=2, source=washpost.com, allow_photos=false, maxitems=7, display_ugc_photos=false, includepause=true, canvas_allcomments_app_instance=6634zxcgfd, includepermalink=false}!!
Comments
SECTION: {section=local, subsection=null}!!!
INITIAL commentConfig: {includereply=true, canvas_permalink_id=washpost.com/8bvh5zpd9k, allow_comments=true, commentmaxlength=2000, includeshare=true, display_comments=true, canvas_permalink_app_instance=m6yzjj840m, display_more=true, moderationrequired=false, includefeaturenotification=true, defaultsort=reverseChronological, canvas_allcomments_id=washpost.com/km4ey0dajm, comments_period=14, includevoteofftopic=false, allow_videos=false, includesorts=true, markerdisplay=post_commenter:Post Commenter|staff:Post Writer|top_commenter:Post Forum|top_local:Washingtologist|top_sports:SuperFan|fact_checker:Fact Checker|post_recommended:Post Recommended|world_watcher:World Watcher|cultuer_connoisseur:Culture Connoisseur|weather_watcher:Capital Weather Watcher|post_contributor:Post Contributor, childrenitemsperpage=3, includeheader=true, includeverifiedcommenters=true, defaulttab=all, includerecommend=true, includereport=true, maxitemstop=2, source=washpost.com, allow_photos=false, maxitems=7, display_ugc_photos=false, includepause=true, canvas_allcomments_app_instance=6634zxcgfd, includepermalink=false}!!!

UGC FROM ARTICLE: !!!

FINAL commentConfig: {includereply=true, canvas_permalink_id=washpost.com/8bvh5zpd9k, allow_comments=true, commentmaxlength=2000, includeshare=true, display_comments=true, canvas_permalink_app_instance=m6yzjj840m, display_more=true, moderationrequired=false, includefeaturenotification=true, defaultsort=reverseChronological, canvas_allcomments_id=washpost.com/km4ey0dajm, comments_period=14, includevoteofftopic=false, allow_videos=false, includesorts=true, markerdisplay=post_commenter:Post Commenter|staff:Post Writer|top_commenter:Post Forum|top_local:Washingtologist|top_sports:SuperFan|fact_checker:Fact Checker|post_recommended:Post Recommended|world_watcher:World Watcher|cultuer_connoisseur:Culture Connoisseur|weather_watcher:Capital Weather Watcher|post_contributor:Post Contributor, childrenitemsperpage=3, includeheader=true, includeverifiedcommenters=true, defaulttab=all, includerecommend=true, includereport=true, maxitemstop=2, source=washpost.com, allow_photos=false, maxitems=7, display_ugc_photos=false, includepause=true, canvas_allcomments_app_instance=6634zxcgfd, includepermalink=false}!!
Show Comments
Next Story
Valerie Strauss | November 14, 2013