A federal appeals court yesterday insulated E.F. Hutton Group from a lawsuit for defamation, ruling that the dispute between Hutton's brokerage subsidiary and its former branch manager in Bethesda is subject to arbitration by the New York Stock Exchange.

The ruling -- the first of its kind in any court -- reversed U.S. District Judge Thomas A. Flannery and extended the reach of self-regulation in the securities industry.

Former U.S. attorney general Griffin B. Bell remains as the sole defendant in a jury trial set for Jan. 19. Flannery ruled in July that the plaintiff, John M. Pearce, could seek compensatory damages from Bell.

The case is an outgrowth of the cash-management abuses that led the subsidiary, E.F. Hutton & Co., to plead guilty in May 1985 to 2,000 felony counts of mail and wire fraud, and of an investigation made for Hutton Group by Bell.

The 2-0 ruling was handed down by a panel of the U.S. Court of Appeals for the District of Columbia Circuit.

Circuit Judge Douglas H. Ginsburg said in the opinion that Pearce was bound by his agreement to a clause in his employment contract with E.F. Hutton that protected Hutton Group but not Bell.

The clause requires use of the New York Stock Exchange's arbitration procedure to settle "any controversy" between an employe and a NYSE member "or affiliate." Ginsburg wrote that Hutton Group was the affiliate of the member.

The exchange defines the procedure to embrace any dispute arising out of the business between a member and "any other person" or "associated person." The judge said, "It is clear that a branch office manager is an 'associated person.' "

Supreme Court nominee Robert H. Bork, a member of the panel when the case was argued in February, "did not participate in the final decision," according to an unelaborated footnote.

Pearce's lawyer, Stephen G. Milliken, said he was undecided whether to seek a rehearing by the panel or by the full appellate court.

Bell concluded his investigation in September 1985 with a report and a news conference. According to Pearce's lawsuit, Bell defamed him by portraying him as one of a small band of employes who had departed from E.F. Hutton policy, thereby becoming responsible for its guilty pleas.