NEW YORK, DEC. 30 -- A study of the October stock market collapse and program trading, sponsored by the New York Stock Exchange, recommends merging the Securities and Exchange Commission and the Commodity Futures Trading Commission.

The study released today, conducted for the exchange by former Attorney General Nicholas deB. Katzenbach, also recommends restrictions on the trading of stock index futures contracts on the Chicago commodities exchanges and the creation of new stock index contracts to trade on the NYSE. The interaction between trading in stocks and stock index futures, often with the aid of computers, has been blamed by some for adding volatility to the financial markets and exacerbating the record 508-point drop in the Dow Jones industrial average on Oct. 19.

Katzenbach's conclusions are certain to invite criticism from Chicago commodities exchange officials, who have been engaged in a war of words with NYSE executives over who deserves blame for the market collapse. If all of Katzenbach's proposals were adopted, one likely effect would be to cut severely into the profit of the Chicago exchanges.

"I think the Chicago market as it is presently designed is a market really built ... for speculation," Katzenbach said, adding that the NYSE is the most important source of capital for American business and encourages long-term investment.

Both the NYSE study and a study released last week by the Chicago Mercantile Exchange have been submitted to the Brady Commission, the task force appointed by President Reagan that plans to submit its recommendations for reform to the White House on Jan. 8.

Although both completed studies were conducted by experts outside the exchanges, they nevertheless reflected the traditional positions of the exchanges on key issues. As a result, Congress is expected to give more weight to the Brady Commission's report to the president.

In recommending that the SEC and CFTC be merged, the NYSE study said the close ties between stock trading, regulated by the SEC, and stock index futures trading, regulated by the CFTC, are "too intricate to divide regulatory authority." While praising the past cooperation of the SEC and CFTC, the study concludes, "The potential for disagreement and divisiveness at some future time simply is not worth the risk when no useful purpose is thereby served."

Chicago Merc officials have said they oppose merging the agencies, arguing that the SEC should stick to regulating stocks and allow the CFTC to regulate stock and commodity futures. With political contributions spread amid the congressional committees that oversee the CFTC, Merc officials have appeared confident they can preserve the CFTC as a separate regulatory body.

Rep. Edward J. Markey (D-Mass.), who chairs a House securities subcommittee, said yesterday that he supports the proposal to merge the SEC and CFTC.

The NYSE report recommends that a merged SEC-CFTC have authority to close the stock and futures markets in times of crisis, a power restricted now to the exchanges and the president. The report also said that a single agency could better police "market manipulation" schemes that involve trading of stocks and futures.

The study was careful to say that program trading did not cause recent market volatility, including the October collapse. Program trading may involve the simultaneous trading of stocks and stock index futures to capitalize on price differences in the two markets. Stock index futures are contracts that give investors the opportunity to bet on the future price movement of broad-based indexes, such as the Standard & Poor's 500.

The study concludes, however, that program trading can result in problems because the stock and futures markets have different purposes. To overcome these problems, the study, which does not rely on quantitative analysis, recommends increasing the initial deposit that investors must put up to trade index futures, as well as raising other capital requirements. The NYSE study also recommends changing settlement procedures for stock index futures contracts so that they settle in real shares, rather than in cash.

These changes, which the Chicago Mercantile Exchange has opposed in the past, could diminish futures trading volume.

In a move that could increase trading volume on the NYSE, the study recommends creation of new stock index contracts, which would give investors the opportunity to buy and sell contracts on the NYSE that represent broad stock market averages. It also calls for using an NYSE computer system to facilitate large "block" trades.