The Supreme Court, avoiding an emotional school prayer controversy, declined yesterday to decide whether the Constitution permits voluntary student prayer groups to meet in public high schools.

A five-member majority, citing procedural problems, threw out a challenge to a Williamsport, Pa., high school prayer group. The action reinstated a trial judge's ruling that allowed the meetings at the school, but did not resolve the broader freedom-of-religion issues raised by the case.

Four dissenting justices said they would have addressed those broader issues and would have permitted such religious meetings nationwide.

In a second ruling on religion yesterday, the justices again split 5 to 4 in ruling that the armed forces have broad power to forbid the wearing of religious apparel.

The majority, in a decision by Justice William H. Rehnquist, ruled that the Air Force could prohibit an Orthodox Jewish captain from wearing a yarmulke while on duty.

The officer, S. Simcha Goldman, a rabbi and clinical psychologist, said the Air Force dress code infringed upon his First Amendment right to freedom of religion and that the military must at least show that wearing the unobtrusive skullcap would undermine military discipline.

Goldman, who was assigned to work at a military hospital in California in 1977, wore the yarmulke without incident until he testified in 1981 at a court-martial and a lawyer for the opposing side complained.

Rehnquist, rejecting Goldman's arguments, said it would be one thing if civilian authorities barred the wearing of religious garments. "But we have repeatedly held that the military is, by necessity, a specialized society separate from civilian society," he said. "The military need not encourage debate or tolerate protest to the extent that such tolerance is required of civilian authorities by the First Amendment."

Rehnquist, affirming a 1984 appeals court ruling, said military officials "are under no constitutional mandate to abandon their considered professional judgment" on the "desirability of dress regulations."

Justice William J. Brennan Jr., joined by Justice Thurgood Marshall in dissent, said the Air Force "failed utterly to furnish a credible explanation" why Goldman could not wear "neat and conservative yarmulkes." Brennan argued that "we cannot distort the Constitution to approve all that the military may deem expedient."

"The court and the military services," Brennan said, "have presented patriotic Orthodox Jews with a painful dilemma -- the choice between fulfilling a religious obligation and serving their country. Should the draft be reinstated, compulsion will replace choice."

Justices Harry A. Blackmun and Sandra Day O'Connor also dissented in Goldman v. Weinberger.

The school-prayer case asked the justices to rule once more on sensitive issues involving prayer in public schools. In 1981 the court ruled that state universities could not bar student religious groups from meeting in school buildings if political or social groups were allowed access.

Congress in 1984 also passed the Equal Access Act, which requires public school officials to give student religious groups the same access to facilities afforded other student groups.

The court, in agreeing to hear Bender v. Williamsport Area School District, was expected to decide whether it would extend the 1981 decision to high schools and to determine the constitutionality of the 1984 law.

But Justice John Paul Stevens said a Williamsport school board member and parent, John C. Youngman Jr., who opposed religious-group meetings at the high school, could not show that he or his school-aged son were harmed by the meetings.

As a result, Stevens said, Youngman lacked legal standing to ask a federal appeals court to block the meetings and the appeals court ruling must be set aside.

The dissenters, led by Chief Justice Warren E. Burger, said Youngman, as a parent, had the legal right to challenge the trial judge's ruling allowing the meetings. "I would reach the issue the court originally agreed to address: whether the First Amendment requires a public high school to prevent a student-initiated, student-led group from meeting during an extracurricular activity period," Burger said.

The broader issue, left unresolved yesterday, is likely to confront the justices again, and if just one of the five who voted to dismiss the Pennsylvania case agrees with Burger next time, the court could render a precedent-setting ruling approving such prayer meetings in high schools.

In a third case, the court clarified a 1978 ruling that said individuals can sue municipalities for civil rights violations by employes when that conduct stems from an official policy. The justices ruled 6 to 3 that a single improper act, if sanctioned by a high-ranking policy-maker, could expose a city to such suits. Brennan wrote the opinion in Pembaur v. Cincinnati.