The Washington PostDemocracy Dies in Darkness

What to know before using school ratings tools from real estate companies


Jack Schneider is an assistant professor of education at the College of the Holy Cross, Mass., the director of research for the Massachusetts Consortium for Innovative Education Assessment and the author of the soon-to-be-published “Beyond Test Scores: A Better Way to Measure School Quality.” His earlier books are “Excellence For All: How a New Breed of Reformers Is Transforming America’s Public Schools” and “From the Ivory Tower to the Schoolhouse: How Scholarship Becomes Common Knowledge in Education.”

Schneider co-hosts the education policy podcast Have You Heard and has written several posts for this blog, including one titled, “If Only Teachers Were Smarter …” He can be found on Twitter: @Edu_Historian

This post is thematically related to his new book.

By Jack Schneider

I like where I live, in New England’s most densely populated city.  My wife likes it, too, as does our daughter, who attends the public school across the street.  Each morning, when we walk her to school, we feel lucky to live where we do, and happy about the education she’s getting.  And when we interact with other families at the school — families that represent the many colors, creeds, and conditions of America — we worry a little less that the nation is coming apart at the seams.

But we wouldn’t have moved here if we had given any consideration to the school rating tools available from real estate companies like Zillow and Trulia.  We wouldn’t have even looked at what their websites deem an “average” school, earning only a 6 on a 10-point scale.  Instead, we’d be scrambling to make our mortgage payment in one of the region’s leafy suburbs.

Just because we ignored the ratings doesn’t mean we ignored the basic question of school quality. Before we put our bid in, I visited the school and took a tour; I talked with the principal and assistant principal.  I took in as much information as I could and, equally important, I dismissed a lot that wasn’t particularly informative. It took a lot of time. It also helped that I’m a professor of education, and my wife is a teacher.

Knowing that not everyone has the time or expertise to do their own school quality reconnaissance, real estate companies have worked to meet market demand by leveraging data. Companies like Zillow and Trulia have taken this a step further, by building school performance data into their website — allowing families to shop for a school and a home at the same time.

It all sounds reasonable enough in the abstract.  In practice, however, such systems threaten to exacerbate already troubling levels of school segregation.

The root of the problem is the reliance on standardized test results.  As research has found, school factors explain only about 20 percent of achievement scores — about a third of what student and family background characteristics explain. Consequently, test scores often tell us much more about demography than about schools. Low-income students, for instance, score lower than their more affluent peers, even if they attend high-scoring schools. High-income students, by contrast, score higher than their less privileged peers, even if they attend low-scoring schools. Thus, while it is true that school quality does matter, it is also true that test scores aren’t a particularly effective way of gauging that quality, particularly when we recall that test scores tell us nothing about the many things we want our schools to do.

Unfortunately, Zillow and Trulia, both of which have partnerships with, rely almost exclusively on test scores for their school ratings. GreatSchools ratings are based on raw test scores, which correlate strongly with socioeconomic variables, as well as test score growth. And for high schools, GreatSchools adds in a “college readiness factor,” which is measured by SAT scores and graduation rates — two more variables that correlate with race and class. Looking at such data, we learn relatively little about the actual work of schools.

But it gets worse.

Trulia, for instance, has built this data set into a tool that allows users to choose neighborhoods by sliding a 1-10 “Schools by Rating” bar. Even if more robust data were used to determine these school ratings, this would still be a problematic practice. Schools have unique strengths and weaknesses that can’t be compressed into a single number; the quality of a child’s educational experience depends tremendously on the fit between student and school; and any set of measures, particularly if they are narrowly defined, can be inaccurate. Yet such a tool is even more troubling when we recall that test score data are often simply demographic data in disguise.

Consequently, when users slide the rating bar up, toward higher scores, they often eliminate all but the wealthiest, and therefore whitest, neighborhoods and towns. Where I live, for instance, when the “Schools by Rating” bar is set to 1 — the lowest possible score — 12 nearby schools appear:

But when the rating is set to 5 — an “average” rating — roughly half of the schools disappear from the map:

When the rating is set to 10 — the highest possible score — all of the schools disappear.  Listed houses remain visible on the map. But which parent would want to live someplace without a single good school?

If absence of highly rated schools impacted every place equally, the problem would be less severe. Prospective home buyers would still be misled about school quality, but they wouldn’t be steered anywhere in particular.

If one zooms out, however, there appear to be plenty of “great” schools nearby — located in the surrounding suburbs.  Particularly dense clusters of “10”-rated schools are located just northwest, in Lexington, and southwest, in Wellesley.

These towns, no doubt, have excellent schools. But are they better than ours? There’s little evidence of that. The main difference, it seems, is that they are whiter and more affluent. According to Zillow, the median price of available homes in Lexington is $925,500. In Wellesley it’s over $1.2 million.

Students in these districts do better on standardized tests — something we could have learned merely by looking at demography. But what is their schooling experience actually like? Do they have teachers who care about and challenge them? Do they feel a sense of safety and belonging?  Are they developing critical thinking skills and becoming engaged citizens? Do they have access to the arts? Are they socially and emotionally healthy?  Perhaps. But relying on the data used to produce the “Schools by Rating” scores, these basic questions remain unanswered.

Yet many parents will assume some legitimacy in these numbers. And they will act upon this information, fleeing “bad” schools and competing for “good” ones. Lower-scoring schools will grow poorer, blacker, and browner, while higher-scoring schools will do the opposite. Such shifts, in turn, will impact scores at those schools, and the cycle will repeat, fostering deeper segregation.

To be clear: this is not entirely the fault of companies like Zillow and Trulia. Americans traffic regularly in bad data, or use otherwise reasonable data in highly problematic ways. Newspapers print rank-ordered lists of schools organized by average test score, as if somehow that tells us about school quality. Parents share opinions via word-of-mouth, often without ever having visited a school. And the state, which bears particular responsibility in all of this, punishes and stigmatizes low-scoring schools despite compelling research questioning the wisdom of such policies.

All of this could be ameliorated by better data systems. And there are grass-roots efforts, being led by groups such as the Massachusetts Consortium for Innovative Education Assessment, to create such systems.

Until we have better data, though, we must take care in the way we use it. Information, itself, is not harmful. But when it is stripped out of context, repackaged, and used to encourage particular behaviors, it can become quite dangerous.

My daughter loves her school. And when I watch her with her classmates, I almost forget about the steadily intensifying segregation that plagues American education — a problem that undermines learning and erodes our democracy. Seeing them together is almost always the best part of my day. But I would have missed it had I placed any stock in the score awarded by, or the assessment that her school is merely “average.”

Here’s more to read:

‘If only American teachers were smarter …’

American schools are modeled after factories and treat students like widgets. Right? Wrong.