'Backtesting' May Not Predict Performance

By Tim Paradis
Associated Press
Sunday, December 9, 2007

NEW YORK -- In investing, like sports, the numbers can be compelling but don't always tell the whole story.

Athletes and stock pickers alike can hit cold streaks or lose their rhythm when conditions change, so a solid long-term record is what really burnishes a reputation. Rookie funds are at something of a disadvantage, knowing that investors like to see a history of performance, so many of these funds will calculate what their results would have been had they been around longer. But while the numbers that come from so-called backtesting can be helpful, investors should remember there is a difference between a scrimmage and a regular game.

"It isn't the same thing as real results," said Russel Kinnel, a mutual fund analyst at Morningstar, an investment research firm based in Chicago. "I think you certainly have to keep it in mind when it is backtested," he said of investors considering young funds.

Even if everyone agreed that backtested figures carried as much weight as the fund's actual performance, investors would be wise to remember the boilerplate statement that's as much a part of mutual fund ads as tags on new mattresses or FBI warnings on movies: Past performance does not guarantee future results.

Backtesting is often used by funds that mirror an index. If a fund follows an index, the thinking goes, it's possible to go back and plot how the fund would have traded by looking at how the index did. But in some cases, even tracing an index back through time can present problems.

Kinnel has concerns about backtesting funds that invest in small-cap stocks, where limited supply and demand can make it harder to buy and sell.

Some poorly constructed models might fail to add in trading costs that over the long term would depress returns.

"There's no reason you can't test for liquidity and trading costs and commissions. But it takes more effort," Kinnel said. "You have to make it realistic. And that involves realistic costs."

It's not likely that funds are trying to use misleading data, analysts say. It can simply be difficult to account for all the variables that could have affected how a fund performed. If a fund is not following an index and is instead run using a strategy, it can be hard to know which moves fund managers would have made.

Mark Labovitz, manager of quantitative research for the Americas at Lipper, an investment analyst in New York, contends that backtesting has its place but said the numbers can lose meaning when they reach back too far.

"You're assuming that all the relationships in the marketplace that you use to presently build your models hold going backward, and that may or may not be true. It's less likely to be true the further you go back. If you start pushing things back two or three years, it can be a little dicey," Labovitz said.

Still, fund companies know that investors rely heavily on past performance -- often too much so.

"Everyday investors pay attention to the returns that are reported and to the extent that backtests play into that, that is what's going to grab their attention. I think everyday investors are rather driven by raw performance in their behavior," Labovitz said.

Not all fund managers believe in using backtesting.

"We made a firm decision not to market on hypotheticals," said Gerald Sullivan, portfolio manager of the Industry Leaders Fund.

When he started the fund, he decided he would wait until he had several years' worth of results to show until he began any serious marketing efforts. The large-cap fund, which invests in both value and growth stocks, showed a return of 8 percent through the end of November, a trailing return of 11.6 percent for the past three years and of 12.7 percent for the past five years.

"It can be confusing to the average investor," Sullivan said of backtested results, adding that he isn't comfortable with them. "It's very easy to predict what will have done well in the past."

© 2007 The Washington Post Company