Merrill Lynch, Pierce, Fenner & Smith Inc., the nation's largest securities firm, has dealt what may be a fatal blow to a three-year-old experiment to chip away at the near monopolies that stock exchanges have over trading in many securities.

Merrill Lynch announced that within a few weeks it will stop making markets in securities that are listed on a stock exchange. In essense, Merrill Lynch--despite its preeminence, considered a Wall Street maverick--said it will stop trying to compete with the New York Stock Exchange and smaller exchanges like the American, Midwest and Pacific.

When a securities firm makes a market in a security, it stands ready to buy stock from or sell stocks to the public at a price that it quotes openly.

Robert P. Rittereiser, Merrill Lynch's executive vice president in charge of strategic development, said the firm believed in the experiment--which was devised by the Securities and Exchange Commission--and stayed with it longer than could be justified. Making markets in the securities was unprofitable and required a heavy commitment of resources, he said.

An official of the Securities and Exchange Commission said that Merrill Lynch's decision to pull out of the experiment hurts but does not doom it. He said he is not aware of any major firm willing to replace Merrill Lynch.

When a stock is listed on the New York Stock Exchange or the smaller American Stock Exchange, nearly all of the trading in that stock is done on the floor of the exchange through a single market maker called a specialist. The specialist firm in a stock is charged with matching buy and sell orders that flow to it from member brokers and also with making an "orderly" market in a stock--that is, buying for its own account when it cannot match a buy order with a sell order and vice versa.

The New York Exchange, by far the nation's most important securities market, long has had a prohibition against any of its member firms buying or selling listed securities for customers anywhere but on an organized stock exchange.

Regional exchanges like the Midwest in Chicago or the Pacific in California long have listed stocks that also are listed on the New York Exchange. In recent years, because these regional exchanges have automated their floors to handle small trades--those involving, say, 500 shares or less--many major brokers direct small trades to them.

But brokers and Securities and Exchange Commission officials say there is little "price competition" between the New York and the regional exchanges, that an order to buy a stock does not get sent to a regional exchange because it may be cheaper there, but because the order can be executed more efficiently.

The exchanges have long argued that their floors generally give purchasers the best price because most of the orders to buy and sell are matched up in a central marketplace and that, because of reporting requirements, buyers and sellers can be assured that the price at which their orders are executed are the best prices available at the time. In the jargon of the securities markets, the exchanges are efficient.

SEC Commissioner Barbara Thomas agrees. Because of the automated link-ups among the exchanges, she said, there is competition. She said it is imperative that the securities markets continue to improve their technology--which will expose more buyers and sellers to one another--but she said there is little need for the off-board trading experiment.

The SEC, under a 1975 congressional mandate to foster the development of a national securities market, ruled several years ago that members of the New York Stock Exchange could make their own markets, or trade off-board, in any stock whose first listing on an exchange occurred after April 1979.

According to one SEC official, the driving force behind the experiment was the belief that "although the markets are efficient, the market as a whole benefits from competition."

But the experiment lured few big securities firms besides Merrill Lynch and Goldman, Sachs, which remains an off-board market maker in a few stocks. The other bigger firms that participated--Shearson/American Express and Salomon Brothers Inc., for example--also confined their activities to a few stocks.

Of the nearly 490 stocks eligible in the experiment today, fewer than 90 are being traded off-board. Merrill Lynch, which once made off-board markets in about 30 stocks, was down to 11 when it threw in the towel. For the stocks being traded off-board, 90 percent of the trading volume still occurs on exchanges.

The experiment was limited to newly listed stocks because, the SEC reasoned, they were securities that had been traded over the counter--where securities firms rather than specialists made markets--and that the initial market makers would be willing to continue dealing in those stocks rather than cede the business to the exchanges as they had to do in the past.

Johnston, Lemon & Co., a local Washington broker and a NYSE member firm, continues to make a market in Geico Corp. stock, even though Geico was listed on the New York exchange in 1980. Patrick Ryan--the chief trader at Johnston, Lemon--said that the experiment never worked because most brokers seldom check the prices offered by Geico market makers--like his firm and Merrill--because the process is too cumbersome.

He said that small brokerage firms may have the luxury to check quotes, but the big firms generally toss small orders in the computer, which routes the order to the market that executes it best, not necessarily where the price is best.

New computer hardware is in place that may make it easier for brokers to check the off-board market makers and the rules will require specialists to check the off-board prices to see where the best price can be had. But Ryan, who was optimistic about making a market in Geico in 1980, is not sure he thinks Johnston, Lemon should stay in the off-board business.

He said it will be expensive to buy the necessary computer hardware to display his stock quotes in a new link-up. Even if Johnston, Lemon spends the money to do so, he said, there is no indication that the firm would get anything more than "second-hand orders"--those the exchange specialists sent his way because they chose not to meet the price Johnston, Lemon was offering.