A special three-judge federal court is considering the withholding from trial of evidence considered crucial to the prosecution of former labor secretary Raymond J. Donovan.

The evidence at issue consists of federal grand jury testimony and interviews of Donovan and other witnesses compiled during special prosecutor Leon Silverman's 1982 investigation of allegations that Donovan had ties to organized crime figures.

The special court met in secret session last month in an out-of-the-way room at the U.S. courthouse here. Marshals guarded the doors to make sure no outsiders wandered in, sources said. Lawyers for both sides in the case, scheduled for trial in a New York state court Sept. 2, were present at the hearing but declined to comment on the proceedings.

The judges have yet to rule, but if the evidence is withheld, it could force prosecutors to consider dropping charges against Donovan and other defendants in the fraud and grand larceny case.

Donovan's lawyer, William Bittman, while not commenting on the proceeding, suggested that the special court may be planning to "set standards" for the further use of evidence compiled during investigations by special prosecutors and independent counsels appointed under the 1978 Ethics in Government Act.

The three-judge panel, whose members are appointed by Chief Justice Warren E. Burger, was established under the law to appoint the special investigators, define their jurisdictions and receive their reports. Six such counsels have been appointed so far, under restrictive mandates limited to federal law. None of the inquiries has produced criminal charges.

Special prosecutor Silverman concluded there was "insufficient credible evidence" to warrant Donovan's prosecution for any federal crime. His records were turned over under seal to the special prosecutor's (now independent counsel's) division of the U.S. Court of Appeals here.

About the same time, Bronx District Attorney Mario Merola's office began inquiries, spurred by several potentially related gangland murders in the Bronx.

In four orders during a two-year period, the three judges of the special court, then headed by the late Roger Robb, turned over dozens of the so-called "Silverman materials" to Merola's prosecutors to "enforce the laws of the state of New York."

Sources said the judges of the special court -- Senior Circuit Judges George E. MacKinnon of Washington, Walter R. Mansfield of New York and Lewis R. Morgan of Atlanta -- are balking at use of that evidence at trial. The judges reportedly are contending that the permission they gave to use the evidence before a grand jury and "to enforce the laws of the state of New York" does not mean that the evidence can be used at a trial.

The Donovan case has been the subject of prolonged pretrial wrangling in which many of the "Silverman materials" have been publicly discussed, listed and summarized. Copies were provided to defense attorneys under a Nov. 19, 1984, order granted by the special court following the Bronx indictments.

One defense lawyer, Stanley S. Arkin, speaking for Donovan and the seven other defendants from Donovan's construction company, said in a recent motion in the Bronx court that it was "clear that the Silverman materials are essential to providing a number of apparently important aspects to the prosecution's case."

Following the Donovan indictment in September 1984, the special court here refused a request by Donovan's lawyers that it take back the Silverman evidence. The Supreme Court refused to consider an appeal.

In a routine motion in February, Bronx prosecutors asked the special court, now headed by MacKinnon, to "authenticate" the documents so they could be offered into evidence at the trial.

The panel treated the motion as if it were a request for fresh disclosure, leading to last month's hearing.

Sources said the judges were critical of the prosecution's plans to use the Silverman evidence at trial and, according to a copy of an order obtained by The Washington Post, are demanding to know why it would it be "a miscarriage of justice" if the evidence were denied.

Asked about the session, the chief prosecutor in the Bronx case, Stephen R. Bookin, said yesterday, "I can't comment." But he said he would not disagree with Arkin's description of the importance of the evidence.