The Justice Department suffered a setback yesterday in its attempt to save the prosecution of a sensitive national security case when a U.S. Court of Appeals panel here denied its appeal to set up special rules for the trial.

Judges George E. MacKinnon and Edward A. Tamm ruled that the government's unsusual appeal was premature becuase the case, involving International Telephone and Telegraph Corp. official Reobert Berrellez, hadn't been decided yet in the lower court.

Berrellez is charged with conspiracy and lying to a Senate committee investigating alleged collusion between ITT and the CIA to defeat Marxist Salvadore Allende in Chile's 1970 presidential election.

U.S. District Court Judge Aubrey E. Robinson Jr. dismissed the jury in the Berrellez trial last October after rejecting a proposed sweeping prosecution order designed to prevent national secrets form being disclosed during the public testimony.

Philip B. Heymann head of the department's Criminal Division, told Robinson then he would seek the uncommon appeal to the higher court because persons shouldn't be given "broad immunity" from prosecution because their case might involve classified information.

Department spokesmen had no comment on the decision yesterday. But officials said Heymann's appeals specialists will now study whether to ask the full D.C. Court of Appeals for a rehearing, take the case to the Supreme Court or go back to Robinson to ask that he modify his earlier ruling.

The legal principle in the case is considered vitally important within the department because the fear of disclosing secrets is a recurring problem in national security-related prosecutions.

The proceution had proposed a protective order that would have let it challenge potentially sensitive evidence before it is presented at the trial rather than after, as is normally the case.

The government apparently fears that defense attorney Patrick A. Wall might uncover still-secret CIA relationships in Latin America during the trial.

Part Of Berrellez' defense is that CIA officers helped prepare Berrellez for the Senate testimony that led to his indictment.

Wall said yesterday in a phone interview from New York that "I'm just going to wait and see what the government is going to do. The ball has been kicked back to them."

Heymann told reporters after his appearance before Robinson in October that it was likely that the government would have to drop the charges against Berrellez if the appeal were not successful. A spokesman later qualified that, saying it depended on how decisive the court's ruling was.

Yesterday's single-paragraph memorandum from the appeals court panel did not address the larger issues Justice attorneys tried to raise in their petition. The appeal was denied on the usual technical ground that Robinson had not finally ruled.

"It [the District Court] has shown a proper sensitivity to the requirements of natioanl security, and piecemeal appeals are not favored... except in rare instances," the memorandum said. "This is not such a case."