U.S. District Judge Charles R. Richey removed himself yesterday from presiding over the pending criminal trial of two high-ranking members of the Church of Scientology. But at the same time, Richey rejected their allegations that he was biased against them.

Richey, whose presiding over Scientology cases over the last two years has become increasingly controversial, said the defendants and their attorneys in the case had engaged in "groundless and relentless attacks" on him.

Saying the pending case against two remaining Scientology leaders charged with stealing government files and bugging government meetings had turned instead into a "trial of this judge," he said it was appropriate for him to voluntarily quit the case.

The case had been scheduled to begin on July 7, but Richey postponed it indefinitely after the two defendants, Morrison J. Budlong and Jane Kember, began their latest attempts to get him removed from the case. It will now go to a judges' committee of the federal court for reassignment and a new trial date

Recusing of federal judges is rare, especially when a judge has been handling a series of cases as long as Richey has been involved in the Scientology cases. He convicted seven of the church's leaders, including the wife of its founder, last fall and sentenced them to prison terms.

The Scientologists have contended that they have tape recorded statements from a deputy U.S. marshal and Richey's courtroom reporter that they say show the judge is biased against them.

According to those alleged recordings, the Scientologists assert that Richey was afraid of the church and thought its members were trying to discredit him. They also contend that the judge wanted to handle the case only for its publicity value, and behave in a bizarre manner by, among other things, eating french fries while hearing one aspect of the case in California.

In his opinion yesterday, Richey said the attempt to disqualify him was the "latest effort in the escalating attack on the court. . . ."

He noted that the law requires that a judge accept as fact any allegations in a motion that he remove himself from a case, but that did not suggest that his "recollection of the events discussed by the defendants is consistent with the allegations contained in their affidavits."

Richey vigorously defended his actions in the case, and said at one point that he questioned "the propriety of the tactics employed" to obtain tape-recorded statements from his court reporter.

As for his eating lunch in a courtroom, he said the defendants had agreed to that in lieu of a delay in the proceeding. "The court's conduct was proper and even polite," Richey added.

The judge accused the Scientologists of abusing the statute that allows removal of a federal judge from a case, saying the misuse of the statute "goes to the very heart and core of the vital independence of the federal judiciary, which is essential to the survival of the republic.

"The time has come for the proceedings in the case to proceed on the merits with the attention of all directed at the real issues in the case," Richey said.

The government had argued that even if true, the allegations against Richey did not require him to remove himself from the case.

The secret tape recordings in the case were made by private investigator Richard Bast, who said he was hired by a lawyer employed by the Scientologists to investigate possible violations of their constitutional rights.