NEW YORK -- Even before Black Monday, many of Wall Street's top executives knew they had helped create a Frankenstein's monster. Their new, sophisticated market instruments, including computerized trading and stock index futures, had been backfiring.

For example, some of the biggest underwriters and securities firms had engaged in new-style ''hedging'' operations that had not protected them, but perversely lost millions of dollars.

There are so many new kinds of securities that even experienced financial men can't keep up with them. Felix Rohatyn, senior partner of Lazard Freres, lists ''index futures, options, letters of credit, off-balance sheet commitments, zero coupon bonds, junk bonds, extendibles, convertibles, subordinated issues -- to say nothing of hedging programs that don't hedge.''

Yet, the big shots on Wall Street did nothing to discourage the use of these tricky devices. With the market on an upward joy ride, few paid attention to the occasional glitch -- until the exception became the rule on Black Monday.

Temporarily, the New York Stock Exchange banned computerized program trading, and the Chicago Mercantile Exchange placed daily limits on index futures trading. That appeared to stabilize activity and reduce volatility. But in this dog-eat-dog greedy environment, any recovery that comes too quickly and too easily will spark demands to go back to business as usual. Already, the stock exchange, as of yesterday, has partially lifted the curb on program trading.

''The trouble,'' says former New York Federal Reserve Bank president Anthony Solomon, ''was that the top managers didn't really understand what these new instruments were supposed to do or how they worked. They left it to these 20-year-olds playing with a computer.''

Can Wall Street ever get back to ''normal''? My impression after a series of talks here is that while Wall Street is now scared of the monster that almost devoured it two weeks ago, it still isn't prepared to act decisively on its own behalf.

''Notwithstanding the high level of concern among senior individuals on Wall Street,'' Solomon said, ''I would be very skeptical that Wall Street as such can initiate reforms to get these instruments abolished. Any changes will have to come from the regulators.''

President Reagan has appointed Nicholas F. Brady head of a commission to examine the Wall Street structure in the wake of the crash. But Rohatyn -- who early spotted the dangers inherent in takeovers financed with junk bonds -- notes that while Wall Street is quick to demand that Washington and other governments ''do something'' to restore confidence, "the financial community here doesn't yet recognize that it is part of the problem.''

For example, one lesson from the 1929 crash was that allowing investors and speculators to buy stocks with a tiny down payment (a small ''margin'') accelerated the collapse of prices. When brokers called for more ''margin,'' they often had to sell out the customer, who lost everything.

That led to stiffer margin requirements: today, a buyer on the stock exchanges has to put up at least 50 percent in cash. But for the stock ''futures'' index trading on the Chicago Mercantile Exchange, the ''margin'' is a mere token.

It should be noted that the federal government's anxiety to speed deregulation of financial markets and institutions provided the opportunity for the creation of some of these gimmicks, to ensure the big swingers a sure thing. Instead, they built in high volatility that swamped normal transactions.

''My industry leaped at the chance, and helped create this Frankenstein,'' Rohatyn said. ''What I think has happened is that the Western economic system is now the hostage of the market, instead of vice versa.''

A special concern is the potential impact of a decline in the huge Japanese stock market. Some experts are scarcely able to articulate their worst-case scenario: a collapse of the Japanese stock market that would leave Wall Street in ruins.

Many observers, with hindsight, say that it was perfectly plain that the Big Board was overpriced when stocks were selling at 20 times annual earnings. But look at Tokyo! There, the price/earnings ratio is 80 to 1. Even if that overstates the case because of different benchmarks and cost-accounting practices in Japan, those prices look far too high. (A complicating factor is the extraordinary inflation of real estate values in Tokyo. Nobody knows to what extent money has been borrowed against real estate to provide cash for speculation on stock markets.)

Japan, Wall Streeters whisper, may now be a financial nuclear bomb ready to go off.