David Howell is executive vice president and chief information officer at McEnearney Associates. Stan Humphries, chief economist at Zillow and architect of the Zestimate, responds to Howell’s criticisms in his own commentary. To read Zillow’s rebuttal, click here.
Some of that information can come from sites like Zillow that offer what’s called an “automated valuation model” — AVM for short — that purportedly present a great estimate of the current market value of millions of homes. It’s cool technology, amassing an enormous amount of information from publicly available sources in one place that is then scrubbed through very sophisticated algorithms to predict value. And all of that is presented in an easy-to-use user interface.
And to its enormous credit, Zillow has done a tremendous job in reaching “top of mind” status with consumers. There’s just one problem: Those predicted values are wildly inaccurate and inconsistent.
As real estate agents, we know that one of our most difficult tasks is pricing a home. That holds true whether we are representing a seller or a buyer. Market pressures change from week to week and from neighborhood to neighborhood. The motivation of the parties is always a factor, as is the condition of a home and those around it. No algorithm, however sophisticated, can quantify the value of a kitchen that was remodeled just before a home was put on the market or a yard that is poorly maintained. It simply isn’t possible for any AVM to predict the value of a home with a level of accuracy sufficient to make a housing decision.
Zillow knows that’s true — and they say as much on their Web site (although you have to dig a bit to find it). They have this to say about their “Zestimates” of value: “The Zestimate is not an appraisal and you won’t be able to use it in place of an appraisal, though you can certainly share it with real estate professionals. It is a computer-generated estimate of the worth of a house today, given the available data. Zillow does not offer the Zestimate as the basis of any specific real-estate related financial transaction. Our data sources may be incomplete or incorrect; also, we have not physically inspected a specific home. Remember, the Zestimate is a starting point and does not consider all the market intricacies that can determine the actual price a house will sell for.”
Yet not a week goes by that we don’t encounter a consumer who is fixated on a particular value for a home because that’s what Zillow says it is. Kudos to Zillow for making this kind of impression on the public — brilliant marketing. But our research shows that, on average, those “Zestimates” are within 5 percent of the actual value of a home just half of the time. (Funny, that’s what its research shows as well.)
As real estate agents, if we got within 5 percent of the value of a home that infrequently we’d be out of business. So if consumers want to base their valuation of a home purchase or sale on what they find on the Internet, we suggest they take out a coin and flip it. Heads — that value is within 5 percent (high or low) or what the home is actually worth. Tails — that value could be 10 percent, 20 percent or more off target.
Beginning in 2010, McEnearney Associates has examined the accuracy of the estimates for property values that Zillow provides — its “Zestimates” of value. This marks our fourth and most comprehensive analysis. We took 500 properties in MRIS, our regional multiple listing system, that were scheduled to settle between March 24 and March 31. During that week, we looked for the Zestimates of those 500 properties. Once the properties settled, we compared the actual sold price to the predicted values on Zillow. And to be as fair as possible, we excluded new home sales from our research because it is highly unlikely that Zillow would have details of the home to be built.
To provide some context, we compared the results of the March 2014 research to that of our September 2012 research. In 2012, we researched 280 properties, and we were able to find a Zestimate for all 280. In the research we just concluded, we were able to find values for all but two of the 500 properties.
Generally, Zillow’s predicted market value is not any better now than it was 18 months ago. The Zestimate is within 5 percent of the actual sales price roughly half the time. In September 2012, the Zestimate was just as likely to be too low as too high; now, it is roughly twice as likely to be too low.
As one might expect with a computer-generated value, there are always outliers. In September 2012, of the 280 Zestimates, the highest was roughly 140 percent of the actual sales prices. The lowest was 82 percent. In the research we just concluded, the highest predicted value was 256 percent of the actual sales price and the lowest was 62.8 percent.
There are significant geographic differences in Zillow’s performance in our current research. They get within 5 percent of the actual sales price just over a third of the time for properties in Washington, and within 10 percent of the sales price less than 60 percent of the time. They are within 5 percent of sales price 43 percent of the time in Maryland and almost 60 percent in Virginia.
Zillow is a bit less accurate for condo and coops than for attached or detached homes. Mirroring the overall results, in all three property types Zillow is at least twice as likely to predict a value that is at least 5 percent lower than the actual value as predicting 5 percent high.
Not surprisingly, those properties that sold for $1 million and more were a little tougher for Zillow to estimate accurately.
It got within 5 percent of the actual price just over one third of the time. It fared much better for homes selling between $500,000 and $999,999, getting within 5 percent almost 60 percent of the time, but for homes selling for less than $500,000 they were within 5 percent less than half the time.
Previously from David Howell: