U.S. District Judge George L. Hart Jr. withdrew yesterday from the criminal trial of 11 members of the Church of Scientology after defense lawyers argued that the trial might include testimony involving him.

Hart's decision came over the vigorous objections of Assistant U.S. Attorney Raymond Banoun that the defense was engaging in "the worst imaginable form of judge-shopping" and a "blatant and deliberate and unprecedented" attempt to interfere "in an insidious manner" with the administration of justice.

The decision by Hart grew out of an earlier case he had tried involving the Church of Scientology. After a hearing of almost three hours yesterday, Hart said that he was withdrawing "very reluctantly" because "my impartiality might reasonably be questioned" despite his own feeling that he could be impartial.

The case was almost immediately reassigned to U.S. District Judge his own problems in trying the case. Prosecutors have noted that Oberdorfer was head of the tax division of the Justice Department from 1961 to 1965, the period during which the government moved to deny tax-exempt status for the Church of Scientology.

The 11 defendants are charged in a 28-count indictment with conspiring to plant spies in government agencies to break into government offices, to steal official documents and to bug government meetings.

Hart's ruling yesterday followed the argument by defense lawyer Leonard Boudin that the judge himself could be called to testify. Boudin noted that one document referred to in the indictment mentions Hart. The document, a letter dated April 16, 1976, from one of the defendants to another defendant, asks him to conduct "a complete ODC (overt data collection) and CDC (covert data collection) on Judge Hart."

The investigation of Hart was ordered, Boudin told Hart, after the judge on April 16, 1976, had suggested to an assistant U.S. attorney that L. Ron Hubbard, founder of the Church of Scientology, should be asked to testify in a Freedom of Information Act suit filed by the church.

"Your honor is in the center," Boudin argued. "It's called the eye of the storm and I really don't see how your honor can proceed here."

Banoun said that the defendants had tried six times to get Hart to withdraw from the case and that they were using their own conduct as a justification for his removal.

Hart said he was not "upset" that he might have been the subject of an investigation by the defendants. "I think it is clear," he said, however, "that many people... wouldn't believe that I wouldn't be upset about it."