Statistics Abuse and Me
Tuesday, May 09, 2006
Hi. My name is Jay, and I am an edstatsaholic.
Good to see you all. It has been two years, three months and 197 days since I last abused an educational statistic.
But this has been a rough few months for me. I have tried to be strong. I did a few stories on Advanced Placement in which I made sure not to say that good scores on AP tests cause higher college graduation rates, since we all know that correlation does not mean causation.
Way to go, Jay. Attaboy.
And I thought I was doing fine, but in March ACT Inc. -- you know, those nice Midwesterners who produce that college entrance test that rivals the SAT -- well, they released this report, "Reading Between the Lines: What the ACT Reveals About College Readiness in Reading." It was about 63 pages. As a journalist it is hard for me to sit still that long to read anything, but I thought it was interesting and I even considered doing a story on it.
I mean, it had some very disturbing information. What would you do if you read this? "Student readiness for college-level reading is at its lowest point in more than a decade." They had this chart showing the percentages of ACT-tested students who met their reading benchmark, a level they consider high enough to handle college, for each year since 1994. They said, "During the first five years, readiness for college-level reading steadily increased, peaking at 55 percent in 1999. Since then, readiness has declined -- the current figure of 51 percent is the lowest of the past twelve years."
Okay, okay. I know what you're thinking. They didn't give us any data before 1994, the change didn't seem that great, and the chart made it look worse than it was by showing not the full percentage range from 0 to 100 percent, but just the narrow range from 40 to 60 percent.
How many times has Jerry warned you about that?
Well, yeah, a lot of times, and I am glad you mentioned Jerry. Some of you know he is my sponsor. I don't know how he does it, but there I was at my computer, thinking about how I might write the first few paragraphs of this story, and I got an e-mail from Jerry.
For you new people, Jerry is Jerry Bracey, more formally Gerald W. Bracey, educational psychologist and research columnist for the Phi Delta Kappan, that fancy education journal. He's my sponsor. He's been working with me a long time. I have had my ups and downs, to be sure, but he has stuck with me through it all. His temper gets to him at times, but I think he is proud of my progress. He just sensed I needed some help with this ACT report.
I got his message right here. Right on the top he says, "Before you run a story . . . " Isn't that thoughtful? A lot of people don't like Jerry. I hear he has had a tough time with some of the universities and big deal editors out there, but he has given as good as he got, and anyway his message was just what I needed. I think he sent it to a lot of other people besides me. Jerry knows a lot of edstatsaholics.
Here is the first paragraph. He didn't pull any punches:
"It is highly irresponsible at this point in time to release a report on trends in student achievement and not present those trends by ethnic group along with changes in ethnic composition of the whole group over time. Even No Child Left Behind knows that. The makeup of this country is vastly changed from what it was. Almost 10 million Hispanics were added to the nation between 1995 and 2005. In 1981, whites made up 85 percent of SAT test takers, in 2005, 63 percent."
And I bet you can guess what he brought up next.
You got it. Jerry's a real bear when it comes to Simpson's Paradox. For you newbies, that is when the aggregate group, in this case all test takers, shows one trend or pattern, but the subgroups, like maybe Hispanics or African Americans, show a different trend or pattern, usually the reverse trend. I am using the definition in Jerry's new book, "Reading Educational Research: How to Avoid Getting Statistically Snookered." Anyway, Simpson's Paradox was really relevant to the ACT report. Here is what Jerry wrote to me:
"I don't know if ACT ignores SAT trends because ETS is a rival or because the SAT trends contradict the ACT results. Here are the demographic changes in the SAT pool. ACT does not report them for its test:
|Whites||85 percent||63 percent|
|Blacks||9 percent||12 percent|
|Asian||3 percent||11 percent|
|Mexican||2 percent||5 percent|
|Puerto Rican||1 percent||1 percent|
|Native American||0 percent||1 percent|
The numbers for 2005 do not sum to 100 because the College Board now uses two categories not present in 1981, when it first started releasing scores by ethnicity, Latino (4%) and Other (4%)."
And then Jerry laid on me the Simpson's Paradox numbers. The national average for the SAT went up only four points between 1981 and 2005, but the average for whites went up 10 points, for blacks 21 points, for Asians 37 points, for Mexicans 15 points, for Puerto Ricans 23 points and for American Indians 18 points.
I see some puzzled looks. I remember having the same problem when I first learned about Simpson's Paradox. How can the national average go up so little while all the subgroups have these big gains? Well, you know Jerry. He never takes anything for granted, and he knows I can be forgetful, so he gave me the explanation one more time:
"All groups have gained, but the lower scoring groups have increased in size. Having lower scoring, even though improving, students make up a larger share of the total pie attenuates the national average. Of course, discussing a falling average due to adding more and more people who score low is quite different from saying that everyone is getting more stupid, which is what ACT is saying."
Okay. We get it. So far so good.
Well, I wanted to be fair, so I sent Jerry's message over to the ACT people, and they replied very quickly. There was a lot of other stuff in the ACT report, and in Jerry's message, about NAEP trends and PISA standings and college grades as reflected by the ACT results, but I will just focus on the main point. Here is what the ACT said about Simpson's Paradox:
"Dr. Bracey suggests that our finding indicating a declining trend in achievement on the ACT Reading Test between 1994 and 2005 results from Simpson's paradox. . . . Between 1994 and 2005, in the total ACT-tested population the lowest-achieving group steadily increased by the greatest percentage (3 percent) while the highest-achieving group steadily decreased by the greatest percentage (5 percent). However, college readiness in reading increased between 1994 and 1999 and then declined between 1999 and 2005 for both of these groups as well as for the total population. Indeed, as we state in the report, the same trend was observed in all racial/ethnic groups except one. So Simpson's paradox does not appear to be occurring in this case."
Yeah, I had the same reaction. They don't tell you much, do they. Like, which groups ARE the lowest-achieving and highest-achieving? How much did the other subgroups change? How much did their college readiness scores go up until 1999, and then how much did they go down? And what was the one racial/ethnic group that did not follow the trend, and what were the numbers for them?
I was almost afraid to show this to Jerry. As I said, he can get a bit testy, but at least this time he didn't have anything to blame me for. So I asked the ACT people to give it another try. Jerry always told me: stick with it; don't give up. And I got more from them the second time.
Here is the ACT Reading Benchmark version of Jerry's chart:
In 1994, participation was 79 percent white, 10 percent blacks, 3 percent Asian, 6 percent Hispanic and 1 percent Native American.
By 2005, that had changed to 74 percent white, 13 percent black, 4 percent Asian, 8 percent Hispanic, 1 percent Native American.
|Whites||79 percent||74 percent|
|Blacks||10 percent||13 percent|
|Asian||3 percent||4 percent|
|Hispanic||6 percent||8 percent|
|Native American||1 percent||1 percent|
So the greatest participation gain was by blacks, the greatest drop by whites, and there was very little variation in success at meeting the Reading Benchmark between 1994 and 2005, with whites up 2 percent, Asians and Native Americans up 1 percent each, blacks down 1 percent and Hispanics down 2 percent. It was as ACT indicated, a small increase by most groups from 1994 to 1999 (up 3 percent for whites, 4 percent for Hispanics and 2 percent for blacks) and then a small drop back from 1999 to 2005, with Asians defying the trend by staying nearly flat that entire period.
Anyway, it was a useful exercise in demanding the exact data, and not letting others make your judgments for you. I was happy the ACT people were so cooperative, and I hope they will be more specific in their future reports. It is true that student readiness, as they measure it, is at its lowest point in a decade, but the change since 1994 is very similar to the flat line we have seen in other high school test surveys, so I would not have made as much of it as they did.
Uh-huh. Some of us have noticed you're not being very specific about your own stories, Jay. Are you sure you haven't been fudging some numbers when we weren't looking?
Well, we're supposed to be looking forward, not backward, right? At least this time I got the facts. Next time I might not even need an e-mail from Jerry to motivate me. It's somebody else's turn, right? I will sit down and hope for the best.