The Federal Reserve Board has rejected an attempt by Saudi Arabian financier Adnan M. Khashoggi to form a bank holding company, citing Khashoggi's refusal to cooperate with investigations of questionable overseas payments by American companies.

Khashoggi's attempt to diversify his banking operations effectively was blocked by the Securities and Exchange Commission and the Justice Department, which have been trying for more than two years to get the Saudi millionaire to submit to questioning.

"Mr. Khashoggi's unwillingness thus far to cooperate with United States law enforcement authorities is not keeping with what the board regards as acceptable conduct for the management of a bank holding company," said a ruling published in the Federal Reserve Bulletin.

Khashoggi had applied to set up a bank holding company based on Security Bancorp of Walnut Creek, Calif., Khashoggi owns 97 percent of the $159 million California bank. Under California law, a bank holding company can diversify into some other types of lending that banks are not permitted to undertake.

Formation of any bank holding company requires the approval of the Federal Reserve Board, which routinely asks other federal agencies to report on applicants.

The SEC and Justice are investigating Khashoggi's involvement in hundreds of millions of possibly illegal commissions, payoffs and kickbacks made by Lockheed Corp., Northrop Corp., Raytheon, Ashland Oil and other American companies to obtain business in the Middle East.

Khashoggi has been identified by several companies and federal investigators as both an alleged conduit and recipient of the payments, which were not properly reported in the firms' accounting.

Federal investigators have tried repeatedly to question Khashoggi about the payments, but have no means of forcing him to testify. Once a frequent visitor in America, Khashoggi has not been seen in the country for at least two years, so federal investigators have been unable to serve him with a subpoena.

Washington attorney James Bierbower, who has represented Kashoggi in the bank holding company manner, was out of town this week and could not be reached for comment.

As recently as February, Justice Department attorneys were negotiating with Khashoggi's lawyers in an attempt to get him to voluntarily submit to questioning, the Federal Reserve decision revealed.

On Feb. 17, when the bank holding company application first came before the board, the matter was deferred for 60 days in anticipation "that an agreement might have been reached in the near future for Mr. Khashoggi to submit to questioning."

Two months later, however, the SEC and Justice told the Fed "the negotiations with Mr. Khashoggi have been treminated and (that) he has not evidenced a willingness to cooperate with the agencies in their investigation."

Federal investigators, who have been repeatedly frustrated in their attempts to pin down Khashoggi's role in the illegal foreign payments, said in interviews that blocking his bank holding company application is one of the few actions that they have been able to take against him.

Khashoggi is the central figure in investigations of payments by American firms to get business in the Middle East. His business empire, said to be worth several hundred million dollars, is one of the largest in the Arab world that does not flow directly from oil. Khashoggi made his money as a merchant, representing American companies in emerging markets, where local contacts were vital.

Lockheed Corp. has reported it paid Khashoggi $106 million in commissions between 1970 and 1975 on sales of TriStar and C-130 planes to the Saudi Arabians.

Khashoggi's main firm is Triad, a Luxumbourg corporation that has been an agent for McDonnell Douglas, Fiat, Rolls-Royce, and British Aircraft.

Khashoggi's U.S. investments included Security Bancorp and a smaller California bank, the Bank of Contra Costa.

While he owns 97 percent of the stock of Security Bancorp, he is not directly involved in the management of the company's subsidiary, Security National Bank.

That made the investigation of Khashoggi's bank holding company application unusual, the central bank decision acknowledged. The agency's inquiry "does not customarily extend to shareholders who do not serve as officers, directors or policy making employes" of the company, the ruling noted, but his control of the stock allowed Khashoggi to hand pick the management.