My column Tuesday argued that Americans overstate how good of a financial investment homeownership is, relative to lots of other places they could invest their money. Once you adjust for inflation, home prices on average have not actually grown very much over the last few decades or even the last century, whereas the S&P 500 has grown a ton.

I received a lot of questions from readers, including many pointing out that an obvious benefit of owning a house — mentioned in the column, but not explicitly accounted for in my annual return calculations — is that you get to live in the house you buy! Don’t you save a lot by not having to pay rent somewhere else?

Maybe in some places, but probably not on average when you think about the opportunity costs of tying up all that capital in a single house. That is, if you rented, you could use the money you would have spent on purchasing a home to invest in another asset that earned a higher return.

The exact numbers will depend on what assumptions you make and what market you’re in, but here’s a simplified, back-of-the-envelope example using nominal dollars (that is, not adjusting for inflation).

Let’s imagine you bought the country’s median-priced house in 1982 for $69,000. I, on the other hand, rented a home for $400 (that’s actually much higher than what the Labor Department estimates the monthly rental value of an owned home was around that time, but I’m trying to be conservative about how expensive it is to rent vs. buy). I also invested $69,000 in the S&P 500.

To keep the example simple, let’s also assume you paid cash for the house –interest rates were about 16% in 1982, after all — and I am paying my rent out of my stock market account each year. Thirty years later, given national housing price changes, you’d be left with a house worth about $213,000, which is a little bit less than today’s national median new home sales price. (Your house is 30 years old after all, and today’s homes are bigger.) On the other hand, looking at how much rents have risen over the same period nationwide and how much the S&P 500 has grown, I’d have about $719,000 in my Vanguard account. And we both had somewhere to live.

This calculation probably understates the financial returns to renting, since it ignores some of the other expenses of homeownership. If you hadn’t done any maintenance or paid taxes on your house for the last 30 years, for example, you’d probably be homeless. If each year you paid 1% in property taxes, and invested 1% in maintenance of the house, you could have spent more than $90,000 (not adjusted for inflation — so even more than that in today’s dollars!) protecting your “investment.”

That’s not to say you couldn’t have gotten into trouble in some cities. In 1982 the west side of Manhattan or the Mission District of San Francisco probably looked like dangerous, unpleasant places to live – but housing values have skyrocketed there over time, and so if you bought there in the 1980s you did quite well. On the other hand, if we were looking at Detroit or Cleveland, buying a home would have been a terrible proposition relative to renting and investing your money elsewhere. It was probably hard to predict ex-ante which cities would do well and which wouldn’t.

I’m not saying buying your home is never a good idea. People derive lots of benefits from homeownership besides shelter and the possibility of financial gain — things like pride, social status, a greater sense of stability and permanence in your community, perhaps access to better schools, not having to deal with a jerky landlord, the flexibility of being able to paint your walls any color you want, etc.

In fact, while Americans believe housing is a better financial investment than possible alternatives, they still are more likely to emphasize “the broader security and lifestyle benefits of homeownership” rather than the “financial benefits of homeownership” when asked about “the best reason to buy a house,” according to Fannie Mae’s latest National Housing Survey data.

The problem is that distortions from the tax code and misperceptions about the financial upside to buying likely cause people to buy too much house relative to what they actually derive pleasure from. And that has real costs to society.