Five years ago, Adrian Antoniu and E. Jacques Courtois Jr. were young hot-shot brokers on Wall Street. Former classmates at Harvard Business School, they had joined Morgan Stanley & Co. and quickly moved up to the mergers and acquisitions department, where the action was the hottest.

Courtois, the son of a prominent Montreal attorney, was named a vice president at age 29; Antoniu, a Romanian immigrant, became a jet-setter and married the daughter of the chairman of Twentieth Century Fox.

While the pair enjoyed the admiration and envy of their colleagues and friends, a few miles away, behind closed doors high above the trading floor of the New York Stock Exchange, a black box was ticking. It would ultimtely lead to their undoing.

The numbers it spewed out alerted surveillance officials to the possibility of unusual activity in several stocks being traded 11 stories below, triggering an investigation that ended two years later with Antoniu sentenced to prison and Courtois a fugitive in Colombia. They and three accomplices were charged with conspiring to use confidential information on imminent mergers and acquisitions to trade stocks, obtaining illegal profits of $600,000.

The case, a particularly complex scheme involving trades in 14 stocks over a four-year period, was eventually solved with the aid of state, federal and international sleuths. But it originated in (and remains the biggest case ever handled by) the New York Stock Exchange's investigatory arm, Market Surveillance Services.

For the past two years, there has been a new emphasis on curbing inside trading, and the NYSE surveillance program is a starting point for many of the investigations.

In a recent interview, Charles G. Ehrhard, head of the surveillance service, described how he and nine colleagues go about tracking illegal insider trading.

The "black box" is actually a series of machines that monitor trading under the collective name Stock Watch. The first device, named a "depth exception review," measures price movements over each 1,000 shares of a given issue. Generally, a given stock is expected to trade within a certain range, based on the past month's performance. For example, if the stock is expected to move up or down just 1 5/8 for every thousand shares traded, and one day it moves 1 3/4, the computer will flag it.

Ninety-nine times out of 100, manager Sheldon Richter explained, it's a question of error, of someone on the floor pushing the wrong key or leaving off the fraction. A call to the specialist usually suffices to clear up the situation. But that one time in a hundred can be an indication something is happening.

A second machine, known as the A and B kickout, flags stocks that exceed two separate parameters, A and B, regardless of volume. If a stock goes up or down by more than 5 percent, as a movement of 5 points on a share selling at $100, the computer will flag it. Again, although this is not an indication that anything is wrong, it is a signal to the surveillance service to check out what is happening.

For example, last month's unusual activity in Texas Instruments--several hundred thousand shares sold below the market price the day before the corporation announced it expected a $100 million quarterly loss--kicked out on this machine. The Securities and Exchange Commission is now investigating to see if the transaction was made by someone in possession of non-public information.

A third machine, the inquiry station, can reconstruct the last five or 10 trades of a given stock to check continuity in trading, how rapidly the stock is moving up or down, and whether an orderly market is being maintained. It can provide a trading profile since the stock opened that day.

After errors are discounted, the computer each day kicks out an average of 40 stocks with irregularities that require further study. The surveillance team then looks for an explanation with the aid of an information retrieval system. This computer carries corporate announcements, analysts' recommendations, and news stories from leading business publications, all of which could affect the price of the stock. A favorable article could cause either the price or the volume of a stock, or both, to increase beyond its customary range.

However, if there is still no apparent explanation, surveillance calls a member of the exchange on the floor to look into the situation. The member keeps an eye on the specialist to see that he can keep up the order flow; i.e., matching buy and sell orders. If trading has gotten out of hand, the official is empowered to halt trading. This occurs about half a dozen times a day for the 1,500 issues listed on the exchange.

The exchange staff telephones the company whose stock is trading irregularly to find out if it has a pending announcement. The staff then gets a print-out of the stock's daily trading record--opening and closing prices, volume and net change--for the past six weeks. (The equipment is capable of pulling the record for as far back as 18 months, but this is seldom necessary because few inside traders could foresee mergers that far in advance.)

The Stock Watch program has been in place since the mid-1960s. Recently a further refinement was introduced. Whereas once every trade on the NYSE tape was tied to just one of the parties and a manual search was required to find the other, now a touch of the button produces a computerized audit trail showing both brokers, as well as the time of the trade, the number of shares, and the price of the stock. The audit trail helps speed any subsequent action.

At this point, Ehrhard's office decides whether to pursue the information provided by the machines to determine if market manipulation or insider trading has occurred.

About 100 times a year, irregularities are handed over to Erhard and his team, although scarcely 10 percent of them turn out to have grounds for disciplinary action. The investigators pick out the period that shows the greatest irregularities and give the chart, along with the Dow Jones averages for those days and any news items, to an NYSE analyst. He or she then contacts the brokerage houses to find out the names of customers doing the trading. That may entail as many as 22 brokerage firms who may traded the stock in question on behalf of 300 customers each.

Since the NYSE has jurisdiction over only employes and brokers, it must call in federal or state regulators to pursue investigations beyond its purview.

When insider trading is suspected, the investigators next set to work establishing an organizational chart similar to a "family tree." At the crown is the company whose stock is the object of the inquiry; at the roots, the stock purchasers. The object is to fill in the branches, to link anyone who bought with anyone who may have had knowledge of non-public information, such as lawyers, accounting firms or banks.

Working with the aid of Who's Who, Standard & Poor's and other reference volumes, they look for people who are directors of multiple companies and who therefore might have leaked inside information. In one recent case, for example, the director of the target company was also a director of an insurance company and a bank. During the period under investigation, the insurance company had purchased 11,000 shares. The bank was located at the same address as a brokerage firm, which had purchased 15,000 shares.

The New York Stock Exchange, unlike the Internal Revenue Service, does not pay bounties to informers. However, it does rely on their information. Frequently, jealous colleagues will tip off the authorities that a person is trading on inside information.

The case of Antoniu and Courtois, the two young brokers at Morgan Stanley, began in the spring of 1978, when Stock Watch indicated unusual activity in a number of stocks. Further examination showed that all were takeover targets and that the trading occurred just before public announcements were made.

After several months of research, the investigators found that in each takeover situation, either Morgan Stanley or Lehman Brothers Kuhn Loeb represented one of the companies involved. They also found that Antoniu had worked for both firms.

But it was two years before the investigators were able to amass enough incriminating evidence to persuade Antoniu to plead guilty. In the interim, his alleged co-conspirator, Courtois, fled to Colombia.

Antoniu and Courtois provided the information about mergers to an American trader and two Europeans who would make the trades and split the profits with them.

In the end, Antoniu was sentenced to three months in prison and fined $5,000, but is appealing his sentence. Newman was convicted of mail fraud and conspiracy in May 1982 and sentenced to a year and a day in jail plus a $10,000 fine. He has appealed his conviction to the Supreme Court.

One of the Europeans, a naturalized Greek named Constantine Spyropoulos, pleaded guilty and received three years probation plus a fine. The government is still trying to extradite the fifth member, a Romanian living in Brussels.

Colombia has granted extradition in the Courtois case, but U.S. lawyers have been unable to find him, according to the SEC.

Last spring, the SEC again asked for stiffer penalties for inside trading. It would like the power to assess civil penalties of up to three times the amount of illegal gains. In criminal cases, it seeks legislation to increase the maximum fine from $10,000 to $100,000.