Zillow's Got Cool Tech and Hot Stock, But Does It Actually Work?

Zillow, the home price voyeurism site, has had a remarkably hot debut on NASDAQ. Priced at $20 per share yesterday, the stock popped to more than $40 in the company's first day of trading.

The company sells subscription services to more than 10 million local real estate people. That brought in $13 million in 2010. Throw in $17 million selling ads on the website in that year and you've got a decent business. Particularly given that the subscription sales are booming right now. The company was even almost profitable last year, posting just a $6.8 million operating loss.

The whole company is built atop what they call a database of real estate information that gets crunched by algorithms to come up with estimates ("Zestimates") for home prices among other things. Want to know if your house is worth more than the Jones's, even if neither of them is up for sale? Just ask Zillow.

"Our algorithms will automatically generate a new set of valuation rules based on the constantly changing universe of data included in our database," the company says. "This allows us to provide timely home value information on a massive scale. Three times a week, we create more than 1.2 million unique valuation models, built atop 3.2 terabytes of data, to generate current Zestimates on nearly 100 million U.S. homes."

Mmmm... Delicious Big Data! But I've got a nagging suspicion that Zillow's numbers -- for all their algorithmic intelligence -- aren't quite as accurate as they'd like us to think. The company flags the issue in its IPO registration.

We may in the future be subject to disputes regarding the accuracy of our Zestimates and Rent Zestimates.

We provide our users with Zestimate and Rent Zestimate home and rental valuations. A Zestimate is our estimated current market value of a home based on proprietary automated valuation models that apply advanced algorithms to analyze our data; it is not an appraisal. A Rent Zestimate is our estimated current monthly rental price of a home, using similar automated valuation models that we have designed to address the unique attributes of rental homes. Revisions to our automated valuation models, or the algorithms that underlie them, may cause certain Zestimates or Rent Zestimates to vary from our expectations for those Zestimates or Rent Zestimates. In addition, from time to time, users disagree with our Zestimates and Rent Zestimates. Any such variation in Zestimates or Rent Zestimates or disagreements could result in distraction from our business or potentially harm our reputation and could result in legal disputes.

But that's really the only mention of this possible problem. A group at the University of Arkansas has tried to assess how good Zillow's information is over the past couple of years. First, they compared Zillow's data to the data from MLS, another real estate site, and found "a large discrepancy in information between the two." More recently, they looked at if data contributed by Zillow users improved things. They found that home entries got more complete but "the accuracy of Zillow's edited facts is not high."

On a purely anecdotal level, it seems like every time I talk with a homeowner (which I'm not) about what Zillow says about her property, she launches into some discussion about how the site gets a bunch of stuff wrong about her home. I'm curious: how would you rate the quality of the information in Zillow's "living database" about your home?

Update, 12:53 p.m.: Zillow keeps their own records of Zestimate accuracy, too, comparing their estimates with actual sale prices all the way down to the county level. It turns out that how close Zillow usually gets to the sale price depends a lot on where you live. In Denver, there's almost a 90 percent change that an actual sale price is within 20 percent of the Zestimate. At the other end of the spectrum is Dallas-Ft. Worth, where there is only a 71 percent chance that the sale price falls within that band around the Zestimate. In any case, good to see Zillow publicizing these numbers.