A federal appeals court panel ruled here yesterday that a U.S. District Court judge erred when he denied a new Abscam trial to former representative Richard Kelly (R-Fla.) without resolving disputed allegations in Kelly's motion.

The motion for a new trial was sent to the judge for further proceedings. But the three-judge appellate panel denied Kelly's request to be released from the prison at Eglin Air Force Base in Florida, where he began serving a six- to 18-month sentence last November.

Kelly's lawyer, Stephen J. Wein, said he was pleased with the ruling but that he had not seen a copy of the court's opinion. Court officials said they did not know when the additional proceedings ordered by the appeals panel would take place.

Kelly, 61, was convicted of bribery and conspiracy in January 1981 after jurors saw a videotape of him stuffing $100 bills into his pockets, part of a $25,000 bribe he was accused of accepting. U.S. District Court Judge William B. Bryant later threw out the conviction but was overruled by the Appeals court here. Bryant then sentenced Kelly, and the conviction was upheld on appeal.

Kelly's motion for a new trial is based on evidence uncovered in 1984 that a former Federal Bureau of Investigation informer contacted Kelly before his trial, discussed defense strategy with him and his attorney and stole documents relating to the trial. The informer said he later turned them over to another FBI informer, Mel Weinberg, for possible use by the government.

Prosectors produced no evidence to support their claim that they never saw the documents. Writing for Judges Antonin Scalia and Kenneth Starr, Patricia M. Wald said, " . . . Some kind of hearing or other evidentiary process was needed to resolve disputed facts." Kelly served in Congress from 1975 to 1981.