The Securities and Exchange Commission is investigating abusive sales practices that it says result from squeezed profits in the brokerage business.

Since the end of the 1982-83 bull market, as securities firms have cut back their costs to improve productivity, it has led to some abuses, SEC Commissioner Aulana Peters told the Securities Industry Association convention here.

Peters said there have been two or three cases of major firms failing to discipline their employes for illegal tactics, such as churning of accounts, unauthorized trades, inadequate disclosure in options trading and disregard of the know-your-customer rule, because these employes generated big commissions. She declined to identify the firms.

Peters noted that brokerages were encouraging such behavior by cutting compensation for low producers and raising it for top producers. For example, one major company reduced its percentage of gross commissions from 34 to 25 percent for salespeople generating less than $100,000 a year. Another firm is rumored to be firing employes who produce less than $250,000 in sales annually.

There is a broad SEC rule requiring corporations to supervise their employes. Peters warned that "in the upcoming economic crunch we may be called up to enforce that rule more frequently than in the past. But if that does not work we must see if modification is necessary," she said, raising the possibility of the SEC's toughening the rule.

The securities industry is coming off its worst year since 1977. Barring an unforeseen boom in the market during December, brokerage firms expect sharply reduced profits in 1984.

The Securities Industry Association estimates the after-tax return on equity for the industry will be between 7 and 9 percent this year, compared with 20 percent last year and 22 percent in 1982.

Profits during the first half of 1984 amounted to $413 million versus $2.7 billion in the first six months of 1983. During the second quarter of 1984, the industry actually suffered a loss of $98 million, its first quarterly loss since 1978.

Despite a stock market rally in August, business has been lackluster, officials said. National full-line brokers suffered a loss during the first half. Commission revenue fell 18.3 percent on an annualized basis because of declines in equity prices and less retail activity. Investment banking, on the other hand, fared better.

During the 1982-83 bull market, both the number of brokerage employes and their compensation increased significantly. This year, on the other hand, has been marked by cost reduction efforts.

In another development, the SIA said it has decided to sue the Federal Deposit Insurance Corp., which recently said it would permit state-chartered banks that are not members of the Federal Reserve system to enter the securities business.

The SIA also said it has formed a committee to work for a universal clearinghouse for different securities markets. Under current rules, investors who buy on margin have to put up double margin when they buy stock and then buy futures or options to hedge against a downturn in the stock. If the trades were integrated, it would tie up less investor money and increase liquidity in the market.

The SIA's outgoing chairman, Richard H. Jenrette, made a plea to members to get more individuals into the stock market. Only 20 percent of all families now own stock, compared with 25 percent in 1977, according to a Federal Reserve study, and institutions account for almost half of the trading volume.

Individuals have left the market because of declining prices, increased volatility caused by institutional trading and a general feeling that they cannot compete fairly against professionals.