It was bad enough that C. Gerald Goldsmith, President and chairman of Intercontinental Diversified Corp. had been authorized in 1972 by the company's directors to make "political and other payments in the Commonwealth of the Bahamas," according to the Securities and Exchange Commission.

But what really surprised SEC investigators was that Goldsmith said he was making the payments, and then kept the money - more than $3 million.

In a complaint filed in U.S. District Court here yesterday the SEC charged Goldsmith and IDC with several violations of the Securities Act regarding the payments.Both, in turn, signed consent agreements to cease the practices cited by the agency and correct previous public filings to reflect the allegations.

And, according to SEC officials, Goldsmith has returned $2.95 million to the company.

Goldsmith left the firm in February 1966, after the SED investigation began.

IDC is a major development company in the Bahamas, owning shopping centers, utilities, hotels, casinos and the Port Authority there. Its stock is traded on the New York Stocks Exchange.

The papers filed by the SEC indicate that Goldsmith laundered the payments through an elaborate maze of seven different bogus firms and the Castle Bank and Trust Co. in Nassau and the Cayman Islands.

In addition, the SEC alleges, several officers of IDC probably violated customers laws by acting as couriers to bring to the Bahamas large amounts of cash from a bank official in Miami who was working with Goldsmith. Carrying as much as $40,000, three different corporate officers did not report to customs, as required by law, that they were carrying excess of $5,000 out of the country.

According to the SEC, Goldsmith would, through various means, divert cash to the Castle Bank accounts. The money would eventually go through Castle into a Canadian bank account of Zodiac Shareholding, Inc., a Panamanian corporation controlled by Goldsmith.

The SEC also charges that Goldsmith arragned for several attorneys to supply IDC with fake legal bills for hundreds of thousands of dollars.

In all, the SEC claims that Goldsmith diverted over $3 million from IDC coffers.

During the course of its annual audit,IDC discovered that its outside auditors had learned of the payment program, and consequently made an effort to substantiate the payments for the auditors.

Goldsmith thereafter produced receipts from four politicians indicating that they had received a total of about $457,000 from Goldsmith, the SEC reported.

"The company has been advised by the four politicians," the SEC says, "that they never received the money and that they gave blank receipts to Goldsmith after he misrepresented to them the purpose for the receipts."

One of the attorneys involved in the phony billings to IDC reportedly began an investigation into Goldsmith's activities for IDC.

This case is considered an important one by many SEC lawyers working on corporate payoff probes. In several cases, they say, they have suspected that officials who have reported payoffs actually kept t he money.

This is the first time they have proven such activity.