Posted at 06:00 AM ET, 03/14/2011

# Gaming the system in D.C. schools

My guest is Aaron Pallas, professor of sociology and education at Teachers College, Columbia University. He writes the Sociological Eye on Education blog for The Hechinger Report, a nonprofit, non-partisan education-news outlet affiliated with the Hechinger Institute on Education and the Media. Pallas has also taught at Johns Hopkins University, Michigan State University, and Northwestern University, and served as a statistician at the National Center for Education Statistics in the U.S. Department of Education. This post is a fictional account of how standardized tests are gamed in the District, where teachers are in the process of prepping kids for the exams, as reported by my colleague Bill Turque here.

By Aaron Pallas

What follows is a fictional account that illustrates an all-too-real truth: school-district leaders and politicians, especially those who stake their careers on rising student test scores, aren’t exempt from engaging in behavior that fatally compromises the validity of the test scores

Marcus is a fourth-grader at Smithdale Elementary, a public school in Washington, D.C. All year, his teacher, Ms. Fishman, has been preparing him and his classmates for the D.C. Comprehensive Assessment System (DC CAS) exam, which is administered annually in the first week of April. The fourth-grade mathematics test is intended to cover the learning standards adopted by the Office of the State Superintendent of Education for the District of Columbia.

There are 53 standards for what fourth-graders should know in mathematics, spread across the following strands: number sense and operations; patterns, relations, and algebra; geometry; measurement; and data analysis, statistics, and probability. In the realm of number sense and operations, for example, fourth-graders are expected to know five different things about fractions, such as the relationships among halves, fourths, and eighths, and among thirds, sixths, and twelfths, with the ability to compare and order these fractions.

The fourth-grade DC CAS math test has 54 items on it, some of which are multiple-choice and others for which students must construct a response. There are a total of 60 points possible on the exam.

Although Marcus’ score on the exam has few direct consequences for him, the fortunes of Ms. Fishman, of Smithdale Elementary, and of the D.C. Public Schools overall will rise or fall on how students perform on the DC CAS. Ms. Fishman’s evaluation as a teacher, for example, is based in large part on how her students do on the DC CAS; if they perform more poorly than predicted, she could be fired.

The overall assessment of how the D.C. Public Schools are doing is also heavily dependent on the performance of students on the DC CAS. If progress from year to year stalls, the package of reforms introduced by former Chancellor Michelle Rhee and sustained by Kaya Henderson, Rhee’s successor as chancellor, are called into question. Conversely, if students continue to improve from one year to the next, these reforms may be judged effective.

On a snowy day in February, Ms. Fishman bursts into the classroom. Barely able to contain her excitement, she swears the class to secrecy. She’s received some inside information, she tells the children. Those 53 fourth-grade math standards? Not all of them will appear on the test! She’s gotten hold of a sheet of paper that indicates only 36 of the 53 standards will be tested, and she knows exactly which ones. Even better, the document indicates that fully one-half of the maximum score on the exam can be achieved by mastering just eight of the 53 standards.

For the next two months, Marcus and the other students in Ms. Fishman’s class study those 36 standards over and over. The other 17 standards are ignored; after all, they won’t be on the test, and neither Marcus nor Ms. Fishman will be held accountable for whether Marcus has mastered them. Ms. Fishman is very familiar with the standards that will appear on the April exam, for the vast majority of them have appeared on the tests over the last few years.

The test days arrive in April, and Marcus and his classmates do their best. In July, the results of the test are released. Marcus and most of his classmates are classified as “proficient” in fourth-grade mathematics. Ms. Fishman’s job seems to be secure, although she won’t be sure until she gets her DC IMPACT evaluation score a few weeks later.

Marcus and his classmates may not know it, but they’re being cheated. Ms. Fishman and the other fourth-grade teachers in the D.C. Public Schools are supposed to be teaching their students 53 different learning standards in mathematics. But that sheet of paper she got in February changed the course of the next two months, shaping what Ms. Fishman taught and what Marcus learned.

Marcus’ performance on the April DC CAS math exam did not reflect his knowledge of all 53 fourth-grade math learning standards. And it’s a huge leap of faith to assume that Marcus’ knowledge of the 36 standards appearing on the exam—which were drilled for two months prior to the April test—can tell us about his knowledge of the 17 standards that were not on the exam and that received little, if any, attention during that two-month period.

And if other fourth-grade teachers in the D.C. Public Schools were to get their hands on that sheet of paper detailing which learning standards would appear on this year’s math test, the problem spreads beyond what individual students such as Marcus might know about mathematics. The DC CAS test no longer does what it is supposed to do: serve as a barometer of students’ mastery of the 53 learning standards adopted by the Office of the State Superintendent of Education as what fourth-grade students are supposed to know about mathematics.

The DC CAS cannot tell us what students have learned about the 17 learning standards that do not appear on the test. It is extremely unlikely that their knowledge of those 17 learning standards, which have largely been ignored for the two months leading up to the test, is as great as their knowledge of the 36 learning standards that Marcus and other students have been practicing over and over again. Because teachers such as Ms. Fishman obtained a document outlining which standards were to be tested, the scores on the test are inflated. They don’t tell us what students really know about all 53 learning standards.

But don’t blame Ms. Fishman. That document causing all of the trouble? It wasn’t obtained by subterfuge, or illicitly. The D.C. Public Schools gave it to the schools and teachers. In 2010, and again this year, the leadership of the D.C. Public Schools deliberately gave to schools and teachers an “operational blueprint” of the learning standards that would appear on the DC CAS exams in April. (In 2010, only 30 of the 53 fourth-grade math standards appeared on the DC CAS exam.) Ace Washington Post education reporter Bill Turque published a link to the 2011 fourth-grade math operational blueprint.

The likely consequence? Scores on the DC CAS may go up, as teachers are able to teach to the test in a very direct way. And, as we know, teaching to the test is not necessarily a bad thing if the content on the test is a representative sample of the broad array of skills and competencies it is intended to measure. But the actions of DCPS leaders undermine the validity of the DC CAS exams as measures of the full range of learning standards. The short-term benefits of rising test scores are shortchanging not only Marcus and his classmates, but everyone who has a stake in a realistic appraisal of how well the D.C. Public Schools are serving their students.

-0-