Potential home buyers, temporarily thwarted by high loan rates, will be back in the market in great numbers by mid-1981, regardless of the cost, a housing expert predicted here this week.

Alan Crittenden, publisher of the Crittenden Report and the most bullish speaker at the 88th conference of the U.S. League of Savings Association, said that as soon as buyers realize that inflation has stabilized, they will be back in the home market -- even if mortgage rates are at a 15 to 16 percent level. Crittenden said he believed this leveling off of the economy could occur within the next six months.

Crittenden also predicted that 1.6 million housing units will be built next year, and that smaller houses would be back in vogue.

The Commerce Department reported this week that housing starts rose only modestly, by 1.6 percent, in October while building permits -- a harbinger of future activity -- plunged preciptiously.

"We really don't have much hope for the next four to six months," said Michael Sumichrast, chief economist of the National Association of Home Builders.

"Starts do not reflect activity as much as permits," he said. "Starts lag behind current conditions and are distorted by the involvement of the government in multi-family construction," he added.

In spite of the numerous new loan variations being offered borrowers to ease their financial burdens, Crittenden says that getting the initial heavy down payment will still be a major obstacle for first-time buyers.