The Securities and Exchange Commission yesterday charged that First Jersey Securities Inc. and Robert E. Brennan, founder and owner of the nationwide broker-dealer firm, illegally earned more than $9.6 million by fraudulently misleading customers about stock prices.

In civil lawsuit filed in New York, the SEC asked a federal court to order First Jersey and Brennan to give back any illegal profits from trading in the stock of three companies. The complaint charges the company deceived customers in more than 13,000 transactions involving the three stocks during the winter of 1982-83.

The companies are the kind of small firms that are the subject of Brennan's frequent prime-time television ads. First Jersey's ads often show Brennan flying his own helicopter and conclude with his personal appeal to potential customers to "come grow with us." Brennan, 41, is also the founder and chairman of International Thoroughbred Breeders Inc., owner of the Garden State Race Track.

The SEC asked the court to appoint a special agent to examine all of First Jersey's stock trading since Nov. 1, 1982, to look for other allegedly illegal profits. If any are found, the SEC said it will ask that they be recovered as well. In previous cases, the SEC has ordered illegal profits returned to customers, or paid to the Treasury if the customers can't be found.

In a statement yesterday, First Jersey denied the charges and called the SEC's accusations "false and malicious."

First Jersey Chairman Brennan said he would sue the SEC and the officials who filed the case against him for "knowingly filing a false complaint." Referring in a statement to a succession of SEC investigations of his firm over the past decade, Brennan said "it doesn't surprise me that the SEC chose Halloween day to initiate this new witch hunt against me and my firm."

The SEC complaint concerns First Jersey's role as the market maker -- the principal dealer -- for the stock of the three companies, Sovereign Chemical and Petroleum Products Inc., Rampart General Inc. and Quasar Microsystems Inc., all traded on the over-the-counter market.

As market maker, doing virtually all of the buying and selling in those stocks, First Jersey was the primary, if not the only, source of price quotations for the shares. SEC guidelines generally permit a market maker to "mark up" or add up to 10 percent, to the price of securities it trades to cover its costs and profit.

The SEC complaint alleges that First Jersey made illegal profits through "excessive" markups and markdowns of up to 124 percent.

One example, the SEC alleged, was the firm's sale of more than 3 million units of Quasar by some of First Jersey's branch offices on or about Jan. 13, 1983. The units, consisting of three shares of common stock and one warrant to purchase an additional share, were sold to the public for $1.50 per unit.

Over the next week, First Jersey offered to repurchase the units, telling customers they could make a "profit" of 25 cents to 50 cents per unit, the SEC complaint said. At least 3 million units were repurchased over this period.

At about the same time, the SEC charged, First Jersey split the units it had repurchased and resold the shares and warrants separately to other customers through different First Jersey branch offices, at an average price of $1.21 per share and $1 per warrant. The SEC said that units First Jersey repurchased for an average of $1.78 each were split up and resold for an average of $4.63 per unit, the SEC said.

The complaint alleges that Brennan and First Jersey in its original prospectus on Quasar and through instructions to the firm's salesmen, made false and misleading statements to customers, allegedly telling them that:The shares and warrants composing the Quasar units could not be split up and sold separately without First Jersey's approval for 90 days after purchase, when First Jersey was allegedly doing so without telling its customers;Customers should make a "profit" by reselling the Quasar units to First Jersey and reinvesting in other securities handled by the firm, when the "profit" involved was determined solely by First Jersey.

The SEC charged that First Jersey made at least $4.7 million by charging markups of up to 124 percent in such transactions in the stock of Quasar Microsystems, which is not related to the Quasar brand name used by Matsushita Electronics of Japan. The SEC made similar charges involving trading in the other two stocks.

Brennan said the SEC's description of his firm's activity in Quasar stock was "totally false. They allege in the complaint that at a point in time, our firm went long bought 3 million units of Quasar. That is in error by at least 2.1 million units. During that period, our firm incurred losses in trading of Quasar securities and our customers made profits."

In an interview, Brennan said the error was "done intentionally" or resulted from "absolute total ineptness. . . . These are the same complaints we've been litigating for 10 years. It's absolute garbage."

Since its founding in New Jersey in 1974, Brennan's firm has grown rapidly, and as of last March claimed it had 90,000 active customer accounts served by more than 1,000 employes in its 32 branch offices throughout the United States.

In addition to underwriting new issues, First Jersey also is a market maker for at least 25 over-the-counter stocks at any one time. The firm buys securities from customers who want to sell them and sells securities from its inventory to other customers, the SEC complaint said.

First Jersey's New York City office tells branch office managers which securities to recommend to customers, the SEC said, and Brennan is often directly involved in those decisions.

Ninety percent of First Jersey's revenue comes from transactions in which it buys or sells low-priced securities for its own account rather than acting as broker in transactions between customers, the SEC said.

Last June, more than two dozen changes in First Jersey's operations were recommended by a special counsel who reviewed the firm's practices under a court order as part of the settlement of an earlier SEC investigation of First Jersey. Most of the changes recommended in the report by special counsel Benjamin Lubin have been carried out, Brennan said.

The special counsel's report noted that First Jersey often holds large stakes in companies whose stock it sells. Such a large position "raises questions concerning the possibility of domination and control of the marketplace, which concomitantly might permit the firm to establish the price levels at which the subject securities" sell, Lubin's report said.

Brennan said yesterday that he regards Lubin's report as a clean bill of health for his firm. Unable to let its "vendetta" against him rests, he said, the SEC has "attempted to breathe life back into that corpse," with its suit yesterday.