The mutual fund business, which many people not so long ago thought would be redeemed into extinction will end 1977 as its best year in quite some time, both in terms of sales and performance.

Mutual fund sales for this year are estimated at $6.4 billion, the highest level since the late 1960s, according to the Investment Company Institute, the major trade organization for the mutual fund industry.

For the first time since 1971, sales will outpace redemptions, which are expected to total just over $6 billion this year.

The major reason for the upturn is an expansion and restructuring of the kinds of securities mutual funds invest in. This is exemplified by the phenomenal performance of the municipal bonds funds in 1977, which had new sales of nearly $2.1 billion, a more than four-fold increase over the 1976 total, as investors were lured by the prospect of non-taxable income in a mutual fund format.

Straight corporate bond funds and others that promise current income and minimal risk alos did well this year.

At the same time, the long-term liquidation trend in the equity funds - those that invest primarily or entirely in the stock market - has continued in 1977. Sales of $2.2 billion were offset by redemptions of $4.3 billion. The desultory performance of the stock market makes this understandable.

However, even here this is good news. Equity mutual funds will almost certainly outperform the stock market averages again this year, as they did in 1976 but had not done previously since 1971. (By contrast, most bank trust departments and pension funds will probably lag the averages again in 1977.)

With the Dow Jones industrial average off about 15 percetn this year, the Standard & Poor's 500 down about 8 per cent, and the New York Stock Exchange index off some 5 per cent, mutual funds that invest in stocks are up about 2 per cent through the end of November, according to Lipper Analytical Services, Inc., which follows funds performance.

According to the Wiesenberger Investment Comapanies Service, which also tracks the funds, for the 12 months period ended Nov. 30, "on average all groups out-performance the best of the market averages" and "those in the maximum capital gains and specialized portfolio categories lead the list by a wide margin over the indexes."

The 2 per cent figure is not much to boast about, and it is less than the estimated 6 per cent increase registered by the funds specializing in bonds. But some of the smaller equity funds, and particularly those that specialize in gold shares, had a very good year, some registering gains of nearly 50 per cent in net asset value for the 12 months through the begining of December.

Because of their number, average size and former popularity, the equity funds still account for the largest part of mutual fund assets - $33.2 billion or 75 per cent of the industry's $44.6 billion asset total in 1977, according to ICI figures. This compares with peak mutual fund assets of $59.8 billion in 1972.

(Counted separately by the ICI are the money market funds, those that invest in short-term money market instruments, which through the roof but have held up in popularity surprisingly well since them. As of October, 1977, these funds totaled $3.2 billion in assets, net down much from their August 1975, peak fof $3.7 billion.)

The statistic that is perhaps most revealing about trends in the industry is the fact that bond funds now account for nearly 60 per cent of total mutual fund sales. In 1969, by contrast sales of bond and income funds totaled only 11 per cent of sales with the remaining 89 per cent in equity and balanced funds (the latter combining stock and bond investments in their portfolios).

Alfred P. Johnson, chief economist for the ICI, sees several marketing trends coming together in the last two years that have helped the industry regain its momentum, including both the expansion of the product mix (the latest wrinkle is option funds) and the emphasis on conversion priviledges - or shifts from one fund to another with the same fund complex - that have offset straight redemptions.

Conversions amounted to $1.5 billion in 1976 and in the first 10 months of 1977 were up 29 per cent compared with the same period last year.

The industry has been able to expand its product mix to give investors a wide choice of products that cover the entire risk-reward spectrum, Johnson said. Meanwhile, the other advance has been to make it clear that if an investor was approching a change in his life cycle or wanted a change in his investment objective, he could do so by converting within the complex, if you judge the marketing strategy by what happened in 1977, you have to give it high marks.

If the stock market rallies and equities make a comeback the product mix will change again, clearly, says Johnson. The ICI economist adds that his personal bet is that the market will be higher next year but won't go out any further on a limb than that.

Meanwhile, he says that funds have positioned themselves to meet a variety of potential changes in financial securities markets, not unlike the restructuring that has taken place in the savings industry generally, with at least one product to meet any investor contingency. And he predicts overall mutual fund sales in 1973 will even out-pace this year's $6.4 billion.