Fun with Statistical Excavation

By Jay Mathews
Washington Post Staff Writer
Tuesday, October 24, 2006; 10:20 AM

I used to be a foreign correspondent. I learned that previously unexplored territory always yielded surprises. A Communist Party secretary in China's remote Anhui province might contradict something about the wheat crop that the party bosses in Beijing were saying. A U.S. Army sergeant on the North Korean border might have a very different count of enemy tunnels than what the Pentagon had released.

Big education studies are like that. The main parts are explored and explained, but there usually are unexpected findings that stay hidden until a clever researcher digs them out. That is what happened last week with Brookings Institution education expert Tom Loveless's latest look at the national and international test results, and their surprising contractions.

I did a story for The Post Oct. 18 on the most entertaining of Loveless's findings. He discovered that countries with the highest test scores in math usually had the lowest level of student confidence in their math abilities and the least interest among math teachers in making lessons relevant to students' everyday lives. That was the eye-catching, contrarian part of his report, but there was more.

For instance, Loveless cast doubt on the widespread belief that states are making their tests easier to pass and getting abnormally high student proficiency ratings so that they can meet the federal targets under the No Child Left Behind act. Loveless looked at the numbers and concluded that states did appear to set hurdles lower than the federal government did in its National Assessment of Educational Progress test. But it was also clear, from looking at test results before No Child Left Behind went into effect, that this had been happening for some time. The new law did not make the states cheat. They had always made decisions that made them look good. State school boards were in charge, and they were not interested in scores that made them look bad.

Throughout the report Loveless warned, as good social scientists do, that his conclusions were tentative. He warned that correlation was not causation and many of his findings might not be what they seemed. For instance, the better results on the state tests may have something to do with student motivation. Teachers focus on the state tests. In some cases the state tests affect a student's chances of being promoted, or graduating from high school. There are, by contrast, no negative consequences for a student who does poorly on a federal NAEP test, so on state tests, students will try harder.

Loveless also looked at what different math tests -- the main NAEP test, a separate NAEP test that measures long-term trends, and the Trends in International Mathematics and Science Study test -- actually tested. He found significant differences in the math subjects chosen for emphasis in each test. In this case, his findings suggested not that the NAEP was too tough, as his comparisons with the state tests seemed to indicate, but too easy.

"Knowledge of fractions, for example, is essential in eighth grade math," said Loveless, a former sixth-grade teacher. "But NAEP contains very few items involving fractions and places an inordinate emphasis on facility with whole numbers. A 1997 U.S. Department of Education study compared the eighth grade NAEP to the leading eighth grade international assessment, TIMSS. Only 13 percent of NAEP items were found to involve fractions; on TIMSS the figure was 34 percent."

One possible reason for students in Singapore doing better on the TIMSS test than students in the United States, but having less confidence in their math abilities than American students, may be that Singapore math standards are closer to the TIMSS standards, and thus higher. "Highly confident American students are doing quite well in comparison to their U.S. counterparts," Loveless noted. "And they feel good about that. But if they moved to Singapore, those good feelings would surely dissipate."

These international comparison have always been tricky. Usually they make American schools look bad. But these days Chinese and Japanese educators are visiting the United States and, to the astonishment of their American hosts, are saying they want to learn from us. Sure, the Asian visitors say, their math scores are higher, but we have more Nobel laureates in the sciences. There is something about the messy, disorganized, haphazard way we educate students that feeds their creativity, and they want that.

This sudden rediscovery of American talent for innovation will likely send people such as Loveless diving into the educational statistics again, looking for stuff they can use. Foreign scholars will search our data and our scholars will search theirs. Hopefully there will be useful surprises for everybody.

© 2006 The Washington Post Company