A federal judge in New York said yesterday that he will keep in effect a court order freezing more than $5 million in profits that the Securities and Exchange Commission charged were made in illegal insider trading in Santa Fe International by investors whose identities, for the most part, are cloaked in mystery.

At the same time, Judge William C. Conner signed an order allowing the SEC to take the unusual step of publishing to-whom-it-may-concern advertisements in the Wall Street Journal and the International Herald-Tribune. The advertisements will tell the mystery defendants that the SEC is pursuing them in the courts and may require them to give up their profits even if it cannot require them to give up their identities.

"You can hold on to your dream of ultimate disclosure," Judge Conner told the SEC attorneys, at the same time noting the difficulty of penetrating Swiss bank privacy laws that so far have shielded the identities of most of the investors whose activities provoked an SEC inquiry. The SEC faces similar problems in an investigation of insider trading in Seagram's unsuccessful takeover bid for St. Joe Minerals.

Also under investigation is possible insider trading in stock options of Marathon Oil Co. Heavy orders for Marathon options were placed just before Mobil Corp. announced its $85-a-share offer for the company.

The SEC's investigation of insider trading in the Santa Fe transaction is complicated by those laws and additional difficulties, including barriers to serving the defendants with court papers. At the center of the SEC's investigations are several transactions in Santa Fe stock and options shortly before the announcement that Kuwait's government-owned oil company planned to buy Santa Fe. That announcement immediately drove up the price of Santa Fe's stock, benefitting purchasers who the SEC believes had advance knowledge of the deal.

Only one of those defendants has been named so far, an investor named Faisal Al Massoud Al Fuhaid. Fuhaid is a Kuwait businessman who once worked as an agent in Kuwait for C. F. Braun & Co., a subsidiary of Santa Fe. According to a Santa Fe official, Fuhaid's relationship with C. F. Braun ended before Santa Fe acquired the company.

C. F. Braun is an engineering company active in development of refinery technolgy and as a designer of nuclear facilities, including power plants, fuel reprocessing and plutonium recovery. Rep. Benjamin S. Rosenthal (D-N.Y.), chairman of the commerce, consumer and monetary affairs subcommittee of the House Committee on Government Operations, has questioned the proposed takeover because of concerns about technology transfers that might be harmful to the United States. He has asked the Office of Technology Assessment to examine that issue.

To reach the other defendants, the SEC brought its action against brokerage firms and Swiss banks that had been involved in the transactions, making them nominal defendants in the case and freezing the money, stock and options that the firms were holding for undisclosed customers.

The SEC had difficulty serving one of the nominal defendants, Lombard, Odier & Cie. Lombard is a Swiss firm, and several types of normal service of papers to parties involved in U.S. legal proceedings apparently are not allowed under Swiss law. The court said yesterday that the SEC could service a Lombard subsidiary in London.

Attorneys for several of the nominal defendants asked the court yesterday to allow them to be dropped from the case, but the judge indicated that such an arrangement must be worked out with the SEC.

According to an affidavit filed by SEC attorney Robert Romano, Lombard has told the commission that its customer or customers have insisted that the firm provide whatever protections are afforded by Swiss law to protect his, her or their identity.

The affidavit also noted: "No unknown purchaser has voluntarily entered an appearance in this case or otherwise identified himself, herself or itself."

The court order allows the SEC to publish its advertisement for four successive weeks in an effort to make sure that the ultimate defendants know of the court action. Such an effort should provide notice to any unknown purchasers who are residents of the United States, Kuwait, Great Britain or Switzerland, the SEC said.

"Attention, certain purchasers of Santa Fe International Corp. common stock or options to purchase Santa Fe Common stock," the advertisement begins. It goes on to describe the SEC action and the transactions under investigation.

"If you are any of the category of customers identified above, you are a defendant in the above entitled action and are required to submit an answer or enter an appearance in the action by Jan. 15, 1982, or a judgment of default may be entered against you," the advertisement warns. "If a default is entered, you will lose any money, stock or options now frozen in the United States."