People on the steps of the Philadelphia Museum of Art view the sunrise  on Dec. 4, 2017. (Matt Rourke/AP)

Come  July 1, Philadelphia will regain control of its public school system for the first time since it was taken over by the state in 2001.

The district has been controlled by the state-appointed School Reform Commission, which was nothing if not controversial. It implemented unpopular “reform” plans that included closing schools, opening and supporting charter schools with little or no accountability, evaluating teachers by student standardized test scores, and a “doomsday” budget that zeroed out funding for things such as paper, counselors, and art and music programs.

The reform commission recently voted to dissolve — at the request of Mayor James Kenney (D) and to the delight of parents and activists who been working toward a return to local control for years. Pennsylvania Gov. Tom Wolf (D) and state Education Secretary Pedro Rivera agreed to the plan late in December, and local control will return this summer. Now Kenney is forming a nine-member appointed school board, a structure that city voters had previously approved.

As Philadelphia residents prepare to take back their school district, they may want to consider some advice from one of the country’s most distinguished education researchers. Below is a letter from David Berliner, Regents’ Professor Emeritus at Arizona State University, with some thoughts about what they probably will be able to accomplish and what they can’t do — lessons that transcend the specifics of Philadelphia. It was written to education scholar and public school advocate Diane Ravitch and posted on her blog. (She and Berliner gave me permission to publish it.)

Some of the letter is technical, but the details are important to understand this observation of his:

…[T]he good citizens of Philadelphia will probably not find whatever important things teachers might be accomplishing in their classrooms reflected in the standardized test scores routinely used in Pennsylvania. Instead, they should now think about other credible ways to judge teacher effectiveness.

Berliner is an educational psychologist who is one of the clearest thinkers in the education world about teaching, teacher education, educational policy and the effects of corporate school reform on schools.His résumé is too long to recite, but among the highlights: He is a former dean of the school of education at Arizona State University and a past president of both the American Educational Research Association and the Division of Educational Psychology of the American Psychological Association. He has won numerous awards and written or co-written more than 400 articles, chapters and books.

Among his best-known works are the six editions of the text “Educational Psychology,” co-written with N.L. Gage; “The Manufactured Crisis,” co-written with B.J. Biddle; “Collateral Damage: How High-Stakes Testing Corrupts America’s Schools,” co-written with Sharon Nichols; and “50 Myths and Lies that Threaten America’s Public Schools,” co-written with Gene V. Glass. He co-edited the first “Handbook of Educational Psychology” and the books “Talks to Teachers, Perspectives on Instructional Time,” and “Putting Research to Work in Your School.”

Here’s his letter:

Dear Diane,

A few weeks ago, I was heartened by your column about the return of the public schools to the citizens of Philadelphia. Since then, I’ve been mulling over four things that I wish I could communicate to them. Perhaps you can do so if you think it appropriate. I don’t know the folks there.

First, it will be difficult for teachers to show that they can turn Philadelphia’s schools into higher-achieving institutions. Teachers may help their students become stronger and more engaged learners, but they probably won’t be able to demonstrate student learning in the way that most people understand it, namely, through higher standardized achievement test scores.

The education research community clearly knows what politicians and the media don’t fully grasp: teachers simply don’t account for much of the variance in standardized tests scores. A reasonable estimate is that teachers account for about 10 percent of the variance in standardized achievement test scores. Research also suggests that outside-of-school factors account for 6 times more of that variance! We even have a Philadelphia based study corroborating these estimates.

In a 2014 Educational Researcher article by Fantuzzo, LeBoeuf, & Rouse, 10,000 achievement test scores from Philadelphia were examined. The researchers used two sets of variables as predictors of students’ standardized test scores. The first set were school-level demographic variables such as race, gender, and degree of economic disadvantage. These are the kind of variables that the Coleman Report first revealed as strong influences on standardized achievement test scores.

My own research, and that of many others, has repeatedly confirmed this truth. In Fantuzzo et al. these kinds of variables predicted 63 percent of the between school variance, quite close to the usual estimate of 60 percent of variance accounted for by demographic variables in students’ achievement test scores. Their data, then, are similar to what was found across nations in PISA and PIRLs, and similar to what other researchers find when school-level demographic variables are put into regression equations to predict variance accounted for.

But what Fantuzzo et al. also did was add student-level variables to their equations. For each student, they knew whether the child was pre-term or low birth weight, had inadequate prenatal care, had a mother who was a teen, had high lead exposure, had a report record of being maltreated, had ever been homeless, and had a mother with less than a high school degree. It wasn’t surprising that each of these variables was a negative predictor of achievement test scores, and similarly, it wasn’t surprising that all but one variable was a statistically significant predictor of the standardized achievement test scores.

When the conventionally used school-level demographic variables were combined with student-level variables into the same equation, something quite different was revealed. The between school variance in reading test scores increased from 63 percent to 77 percent of the variance accounted for in the students’ standardized test scores. This leaves us with the task of trying to estimate what accounts for the remaining 23 percent of the variance in these standardized achievement test scores. We can separate that remaining variance into variance accounted for by error in the measurement system (a reasonable estimate might be about 10 percent) and school effects that are independent of teacher effects (which may also reasonably be estimated to be about 10 percent). Now we can account for about 97% of the variance. So, what percent of the variance in student test scores remains for teachers to affect? The answer is clearly almost nothing!

This all suggests that the good citizens of Philadelphia will probably not find whatever important things teachers might be accomplishing in their classrooms reflected in the standardized test scores routinely used in Pennsylvania. Instead, they should now think about other credible ways to judge teacher effectiveness.

Second, it won’t be Philadelphia’s newly taken-over schools that demonstrate how to get greater achievement from the students they serve. Schools, like teachers, may also be doing great things. But as noted, they usually affect about 10 percent of the variance on standardized achievement tests. Given Philadelphia’s high poverty rates, the variables associated with poverty may well account for most of the variance in citywide test scores, leaving little variance for the schools to effect, similar to the teacher effects just discussed.

The third point addresses the obvious question, what then should we do to better understand how teachers and schools effect local student outcomes? Usually we measure this via a standardized achievement test score, but as noted, that is quite likely not to be adequate for those purposes.

Instead, let’s try something different, and use the funds ordinarily paid to a test publisher to train selected parents who, alongside school principals or teacher leaders, could routinely observe classrooms and assist in monitoring the quality of instruction. Parents, principals, and teacher leaders can better learn to evaluate the artifacts of teaching, among them teachers’ tests and students’ answers. Those classroom tests will show both teachers’ understanding of and instructional alignment with the desired curriculum (as evidenced in the test’s items). And those tests will also show the quality of teaching that curriculum (as evidenced in the students answers to the items).

These assessments can be conducted as informal observations, or can use some of the of the more systematic and frequently used observation scales, such as those found in Danielson’s and Pianta’s systems. Scriven’s duties-based evaluation approach is certainly worth trying. And Meier’s and Knoester’s recent book offers a half a dozen other ways to assess students and classroom practices that don’t rely on standardized achievement tests. The point is this: assessing the quality of teachers and schools can be done without using standardized achievement tests that are known to be highly insensitive to what teachers and schools accomplish.

My fourth bit of advice is directed at those who serve on the newly constituted school board. It’s to remember that education outcomes are the result of much more than education polices. The school board will not achieve their improvement targets until other city and state policies better address the needs of Philadelphia’s schoolchildren. Housing policies must be strengthened to eliminate segregation by race and income, and affordable housing is needed to minimize residential mobility, which impedes school achievement. Physical and behavioral health policies must be strengthened, and nurses, counselors, and social workers must be sufficiently resourced to ensure that students attend school daily and have the supports needed to fully engage in learning. Policies to assure food security are needed so students aren’t preoccupied with hunger, and fair wage policies can provide income security for working families overwhelmed by living in poverty. And so on.

Getting their schools back is good for democracy in the city that played center stage in the founding of our nation’s democracy. Getting those schools to function well enough so its students can take on the role of stewards of our democracy is a whole other matter. I hope this advice helps them to do just that.

David Berliner

(Correction: Fixing spelling of John Fantuzzo)