The U.S. Court of Appeals, in a rare public display of hard feelings among its members, yesterday denied a request that the full court reconsider an August ruling upholding the dismissal of a Navy enlisted man for homosexuality.

The action by a majority of the 11-member court drew a pointed dissent from four members of the court's liberal wing, which in turn evoked an uncommonly caustic response from the conservatives.

The dissent, written by Chief Judge Spottswood W. Robinson III, branded the court's August decision, written by Judge Robert H. Bork, a "particularly inappropriate" attempt to "wipe away selected Supreme Court decisions" and accused Bork of conducting "a general spring cleaning of constitutional law."

Bork retaliated by describing Robinson's dissent as "rhetorical excess" couched in "rather extravagant terms" that "overlooks both what we actually did and the necessity for it."

The August ruling, by Bork, Circuit Judge Antonin Scalia and U.S. District Court Judge David W. Williams of Los Angeles, upheld a Navy regulation forbidding homosexuality in the service. Both Bork and Scalia are conservatives appointed to the court by President Reagan.

James L. Dronenburg, a 27-year-old petty officer at the time, was a Korean linguist with a top security clearance when he was accused by a seaman of engaging in repeated homosexual activities in a barracks at the Navy's language school in Monterey, Calif.

After his dismissal by the Navy, Dronenburg sued, claiming his ouster violated constitutional guarantees of privacy and equal protection of the law. But the three-judge panel disagreed, saying, "Private, consensual homosexual conduct is not constitutionally protected."

Bork said the panel could find no unified principle in several Supreme Court privacy decisions that would allow the appellate court to throw out the Navy rule regarding homosexuals. A 1976 Supreme Court decision on privacy and homosexuality was issued without comment, leaving the issue unclear.

"Instead of conscientiously attempting to discern the principles underlying the Supreme Court's privacy decisions," Robinson's dissent said, "the panel has in effect thrown up their hands and decided to confine those decisions to their facts."

Robinson, joined by Judges Patricia M. Wald, Abner J. Mikva and Harry T. Edwards, accused the panel of "an abdication of judicial responsibility" and said it had provided "patently inadequate justification for a ban on homosexuality in the Navy."

The dissenters suggested Bork and Scalia had implicitly criticized the Supreme Court by saying they could find no clear thread in the high court's various decisions on the question.

"Unless the dissent believes that we are obliged to dissemble, enunciating a unifying principle where we think none exists," Bork said, "then its only criticism must be with the adequacy of our analysis rather than our bona fides.

"That criticism, we may note, would be a good deal more persuasive if the dissent set forth (as it conspicuously did not) the unifying principle that we so obviously overlooked."

Bork said the "judicial hierarchy is not, as the dissent seems to suppose, properly modeled on the military hierarchy in which orders are not only carried out but accepted without any expression of doubt."

Both Bork and Scalia, who joined in Bork's comments yesterday, are often mentioned as potential Reagan nominees to the Supreme Court.

The appeals court is generally considered split, 5 to 5, between liberals and conservatives, with Judge Ruth Bader Ginsburg viewed as a swing vote. Judge Malcolm R. Wilkey, a conservative, said Thursday he will leave the court in December.

A 12th position was approved earlier this year by Congress, but has not been filled.