A federal appeals court today reversed a $16.5 million antitrust judgment against American Telephone & Telegraph Co., ruling that the communication giant did not use monopoly powers to keep another company from competing in the business-telephone equipment market.

The appeals court ruling seems to have bearing not only on a major antitrust suit pending against AT&T in the federal district court here, but also on the federal government's attempt to break up AT&T on grounds that it is a monopoly and has used its monopoly powers to impede competition.

The appeals court threw out five of the six claims against AT&T, its manufacturing subsidiary Western Electric and the Southern New England Telephone Co. and sent the sixth back to the federal district court for a new trial.

The court said Northeastern Telephone Co. had "presented some evidence" to demonstrate that AT&T had an anticompetitive purpose in its designing of a protective-coupling device to go between the non-AT&T switchboard and the AT&T phone lines. But the judges said the jury "could not differentiate with precision" among the six types of behavior and said a new trial should be held concerning the protective-coupling device.

The three-judge panel threw out the verdicts that AT&T used pricing, advertising, product introduction, marketing and the local AT&T subsidiary's utility monopoly to thwart Northeastern's sales of switchboards and smaller telephone systems to businesses.

Bruce Kuryla, vice president and cofounder of the nine-year-old Northeastern Telephone Co., said, "We think the decision is erroneous. We are going to appeal to the Supreme Court." Northeastern, based in Milford, Conn., still is selling business telephone equipment, Kuryla said.

AT&T said it was "gratified that the verdict was overwhelmingly rejected" and that the court had "affirmed the principle that a large firm has the right to compete vigorously against any other company in the marketplace."

The judges also reversed the award of nearly $748,000 in attorney's fees for the plaintiff, Northeastern Telephone Co., which filed suit against AT&T in 1975.

Litton Industries has filed a similar, although much larger, antitrust suit against AT&T in federal district court here. Litton is seeking nearly $600 million in damages, which would be trebled under federal laws.

The federal government also has sued AT&T in Washington on monopoly and antitrust charges and is seeking to break up the giant telephone company.

Both the Litton and the government trials are in progress.