A federal appeals court overturned a 1983 malpractice verdict against a Fairfax County lawyer yesterday and ordered a new trial on allegations that the man was negligent in handling a child custody case.

The 4th U.S. Circuit Court of Appeals held that U.S. District Judge Albert V. Bryan Jr. of Alexandria was wrong when he barred trial evidence that might have shown whether the alleged negligence affected the outcome of the custody case. "It would be a fundamental error to allow the judgment to stand," the appeals court said.

A Colorado man, Stephen M. Stewart, hired McLean lawyer B. VanDenburg Hall in 1981 to help him win custody of his two children from his former wife. In his malpractice complaint, Stewart said Hall promised to resolve a Colorado judgment against him for falling $2,400 behind in his child support payments, but never did so.

Stewart alleged Hall also vowed to try to have Stewart's former wife cited for contempt for denying him visitation rights, but did not follow through. A domestic relations court denied Stewart's custody request and cited the $2,400 judgment as a factor.

In a November 1983 trial in federal court in Alexandria, a jury awarded Stewart $20,000 in compensatory damages and $50,000 in punitive damages. The award was thrown out by yesterday's appellate ruling.

The appeals court said that before damages were awarded, it should have been determined whether Hall's alleged negligence had a bearing on the denial of custody to Stewart. "Judge Bryan repeatedly ruled that he was not going to retry the custody case," Hall said yesterday.

The appeals court said the trial had improperly proceeded on the theory that damages could be awarded for negligence even if negligence had played no role in the child custody hearing.

The Virginia State Bar, the dsiciplinary branch of the State Supreme Court, said yesterday that the outcome of a 1982 complaint filed by Stewart against Hall was confidential, but that Hall was "in good standing" with the bar.