The U.S. Court of Appeals here ordered a new trial yesterday over the contention that the Redskins' exclusive lease for professional football in RFK Stadium violates antitrust laws.

The ruling reverses a jury finding that the lease, which says only the National Football League Redskins can play professional football in the stadium until at least 1990, is legal. The reversal is based on the appellate court's finding that the judge in the jury trial improperly instructed the jury on applicable antitrust laws and improperly kept out certain evidence during the case.

The appellate decision comes in a case that was originally filed 11 years ago by a group of local businessmen who were attempting to place an American Football League expansion team in the Washington area in 1965. The businessmen are still seeking damages under antitrust laws, saying the restrictive covenant in the lease was the main reason they failed to get a franchise for Washington.

The AFL has since been merged with the NFL.

Yesterday's decision is the second time that the appellate court has reversed U.S. District Judge William B. Jones in the 11-year-old case. In 1972, the appellate court threw out a ruling by JOnes that the D. C. Armory Board's leasing of the stadium to the Redskins was a governmental action that could not be challenged under the antitrust laws, and ordered the case sent to trial.

The appellate court noted yesterday that Norman Hecht and the other businessmen who were seeking the AFL franchise were "whipsawed" by the position in which they found themselves as they attempted to get the franchise.

"The Redskins would not seriously negotiate for Hecht's use of the stadium unless Hecht had an AFL franchise; the AFL would not seriously consider Hecht's application for a franchise unless he had the use of RFK Stadium," said U.S. Circuit Judge Malcolm R. Wilkey for the three-member court.

The Armory Board, meanwhile, had said its hands were tied by the lease but would accept any agreement between the Redskins adn Hecht concerning their sharing the stadium.

The judges said Jones instructed the jury wrongly when he told the panel that the relevant geographic market Hecht was trying to enter was the national football market, and that the criteria should instead have been based on the Washington metropolitan area.

Also, the judges said, it was up to the Redskins to prove that two professional football teams could not survive in Washington. At trial, Jones said it was up to Hecht to prove that both teams could survive.

Hecht also had asked Jones to rule that RFK stadium was an essential facility for professional football here for the purposes of antitrust law, but Jones should provide the jurors with special verdict forms because of the complexity of antitrust matters, rather than eliciting a more general verdict as he did in the last trial.