New York Stock Exchange Chairman John J. Phelan Jr. yesterday warned the nation's institutional investors, which control billions of dollars, to exercise self-control over how they use their new electronic trading strategies or face the prospect of another market plunge.

On Black Monday, Oct. 19, Phelan said, he watched as institutional money managers dumped dozens of stock portfolios, each worth between $25 million and $50 million. The flood of sell orders, many tied to computerized program-trading activity, overwhelmed the market and its human and electronic systems.

"The big question mark right now is whether institutions have learned from the October experience," Phelan said. On the day of the market plunge, he said, only about a dozen institutions were trying to sell all at once, but he said he feared that in the future it could rise to 50 or 60 institutions.

"We have got to find a way to slow down that momentum," he said.

Institutional managers for pension and other large funds, Phelan said, need to understand that "there is no instant liquidity in times of severe stress." Unless the managers refrain from trying to instantly sell millions of dollars worth of shares, it could lead to a "repeat of last October," he added.

Phelan came to Washington to testify before the Senate Banking Committee, which is looking into the causes of the stock market collapse. He elaborated on his views at a luncheon with editors and reporters of The Washington Post, and also met with White House chief of staff Howard H. Baker Jr.

Stressing the need for institutional investor restraint, Phelan compared the new electronic trading systems that link the stock, options and futures markets to a car that can go 120 miles an hour. "Just because we have the technology to do something doesn't mean we should do it without regard to its impact on the marketplace and its participants," Phelan said.

Asked how he felt about institutional investors trading in stock index futures as a way to speculate on the direction of the stock market, Phelan responded: "It's going to destroy the market."

Program trading is a computer-based strategy that allows the large brokerage firms, acting for their own accounts or for clients, to take advantage of discrepancies in the prices of stocks traded in New York and futures contracts on stock indexes traded in Chicago. Computers tell the traders when to buy and sell in order to lock in their profits, and other computers speed the trades through the exchanges.

"Speculation is beginning to overwhelm the other traditional values," said Phelan.

The chairman said that the NYSE recently determined that 9 percent of 300 institutions surveyed used strategies tied to futures and options. Two years ago, he said, the figure would have been 1 percent, showing that there was "exponential growth" in the number of institutions using these strategies.

Although the securities industry took a benign attitude toward program trading before the market plunge, the climate has changed dramatically.

The presidential task force headed by Nicholas F. Brady and the report of the Securities and Exchange Commission both assigned part of the blame for the depth of the October plunge on programs.

Phelan said that he met this week with 25 securities firms that do 85 percent of the business on Wall Street and was told that almost all the firms wanted to prevent program traders from using the NYSE's computerized order entry system to execute their programs.

In an effort to dampen the wide price swings that accompany program trading, the NYSE voted Thursday to close the computer system to trading programs on any day that the Dow Jones industrial average moves 50 points. There have been 26 such 50-point days since Jan. 1, 1987.

A wave of public condemnation of program trading has washed across the securities industry, Phelan said. He noted that Shearson Lehman Hutton Inc. had announced that it no longer would do programs for its own account. However, it will continue to do them for clients.

Goldman, Sachs & Co., a major Wall Street firm, announced yesterday that it, too, would suspend program trading for its own account. "Goldman Sachs shares the concern that undue volatility is adverse to the best interest of investors and equity markets, ... " the announcement said. Goldman, which will continue the service for clients, said it had not traded heavily for its own accounts.

Other views expressed by Phelan included these:

He favors the appointment of a single regulator for all equity-related products, but does not believe that move is politically possible. He sidestepped questions about whether he favored the proposal of SEC Chairman David S. Ruder to give the SEC additional authority over futures contracts on stock indexes, now regulated by the Commodity Futures Trading Commission. In the meantime, Phelan favored closer coordination between the SEC and CFTC.

The NYSE and the SEC are both concerned about reports of violations of the short-sale rule. The rule states that one cannot sell stock short (selling by using borrowed shares) unless the previous sale has moved the price higher. The rule was adopted some years ago to prevent traders from profiting by driving stock prices down. Phelan said that, as yet, the investigators had not found proof of a violation.

The October experience, Phelan said, was "not a big event" outside of New York and was quickly being forgotten. If the market's plunge is going to have an economic impact, he said, it has not yet shown up. "It didn't cause a major crack," he said.