Site's Alluring Facade Is Built on Shaky Foundation

By Leslie Walker
Thursday, February 16, 2006

When I saw a demo of the real estate service last month, it struck me as so obvious I wondered why no one had done it before.

Then when Zillow launched on the Web last week, I realized why.

Offering automated property valuations via the Internet turns out to be much harder than it seems -- especially if you expect them to be accurate. But after running extensive tests on this ambitious national real estate service, I found it to be so inaccurate that it's not useful.

It's hard to quibble with the company's goal -- "free, instant valuations and data for 60,000,000+ homes." You type in any address, and in most cases Zillow will spit out a free estimate of the property's market value.

But appraisers questioned whether consumers will have any idea how off-base Zillow's free valuations can be.

"What scares me is the consumer who goes out there and makes a decision based on that data," said Richard Powers, president of the Appraisal Institute, the nation's largest appraiser association with 21,000 members. "Consumers really have no way to judge the accuracy of the estimate -- that really is the problem."

Powers said his board members have had mixed results on tests they've been running since Zillow's public beta test went live. "In some areas, we found the results were fairly accurate to the value of the home. In others, we found results that were at least 40 percent wrong."

Zillow president and co-founder Lloyd Frink said the free, advertising-supported site doesn't aim to replace home appraisers or real estate agents. "It is meant as something to help buyers and sellers start a conversation."

How does Zillow work? It starts by buying massive amounts of real estate information from commercial data collectors -- home addresses, tax assessments, square footage, lot sizes, number of bedrooms, prior sales prices and the like. Then its computers identify similar homes that recently sold in the same neighborhoods and use mathematical models to compare their traits and create a "Zestimate," or "estimated market value."

Below its estimate, Zillow shows a mesmerizing array of other data, including features describing the house, a chart of estimated value over the past year and list of similar houses sold recently. You can call up scrollable neighborhood maps with images of the homes, with their values superimposed.

So far, Zillow has identified about 60 million homes and collected enough data to offer estimates on slightly more than 40 million, with more being added daily. Coverage is spotty to nonexistent for the District, Alexandria and Arlington, but plentiful for the Maryland suburbs and Fairfax, Loudoun and Prince William counties.

Trouble is, because Zillow is loaded with dirty data in some places and missing key factoids in others, its Zestimates often miss the mark -- sometimes so widely that I fear that anyone trying to buy or sell a home could get burned by relying on Zillow.

In my own random tests of dozens of local properties, I found about half of the estimates to be sharply off -- more than 10 percent off the actual recent sales price or what someone knowledgeable about the property deemed its market value to be. Many were off by 20 percent or more.

I asked a Virginia title company to randomly pick houses that went to settlement in the past week -- too soon for Zillow to have picked up those market prices -- and asked an agent to identify recently listed homes, along with comparable recent transactions, using the multiple listing system run by the Washington area's Metropolitan Regional Information System.

I found Zillow sharply undervalued one home in Fairfax Station that had sold twice in the past two years, first for $560,000 and then for $630,000. Zillow pegged its value at $499,206. Oddly, Zillow not only had the first sales price in its database but also showed it right below the flawed estimate. Zillow made the same error on a Lorton home, pegging its value at $784,793 while noting it had sold last summer for $900,000.

Those estimates were likely due to a glitch in Zillow's valuation formula, which has been overriding actual sales data too frequently, according to Stan Humphries, Zillow's director of advanced analytics. He said the formula is being adjusted to give greater weight to sale prices.

Other flawed Zestimates may have been caused by stale data on the size, age and other characteristics of houses, several of which had been remodeled. For instance, a new Lorton home on the market for $899,900 was valued by Zillow at only $394,240. I'm guessing it was because Zillow's description was missing one bathroom and two bedrooms.

Zillow offers a tweaking tool called My Zestimator that allows you to fine-tune estimates by adding data about a home, such as the size and cost of renovation work -- and then view, but not save, the modified value. I was able to boost the value of remodeled homes that way, though not enough to match the sale prices.

Zillow runs extensive analyses to calculate its own accuracy rate by comparing actual transactions as they occur with the automated estimates provided by its computerized valuation system. Nationwide, 62 percent of all Zestimates fall within 10 percent of the selling price, according to Zillow. That means 38 percent are more than 10 percent off the mark, which strikes me as significant.

Accuracy runs higher in urban areas where Zillow has more sales data and drops in rural areas where the data tends to be sparser. But the company's track record can be dismal in some metro areas, such as Baltimore, where the company's own accuracy chart shows more than half its Zestimates are 17.7 percent or more off the selling price.

Frink said he considers Zillow's national accuracy rate strong and isn't troubled by its misses. Zestimates, he said, shouldn't be considered replacements for appraisals, which are created by trained, certified professionals who go to a home to verify its traits and physically view comparable homes.

Frink predicted that Zillow, which received plenty of media hoopla when it debuted, will improve gradually as the company adds housing data and tweaks its algorithm for automated value calls.

But until Zillow gets a lot smarter, I don't recommend making it a trusted bookmark.

Leslie Walker welcomes e-mail

© 2006 The Washington Post Company