A divided Supreme Court declined yesterday to decide whether a state university violates the Constitution by denying formal recognition to an organization composed largely of homosexuals.

Justice William H. Rehnquist, one of three dissenters, chided the six majority justices for yielding to "a natural tendency on the part of any conscientious court to avoid embroiling itself in a controversial area of social policy unless absolutely required to do so."

The court let stand a ruling that the University of Missouri violated the guarantees of freedom of speech and assembly when it refused a meeting place on the Columbia campus to the Gay Lib Organization, which says its primary purpose is to provide a forum for a "dialogue" between homosexuals and heterosexuals.

The heart of the university's case was its contention that to provide an on-campus forum likely would incite "imminent violation" of a Missouri law -- similar to laws in most states --that makes sodomy a felony.

The 8th U.S. Circuit Court of Appeals ruled 2 to 1 in June that law violations were not among the organization's purposes, and that only a much stronger showing of "imminent lawlessness" could justify the university's restriction of First Amendment rights.

Agreeing in a separate opinion, Judge William H. Webster, now the director of the FBI, saw "absolutely no evidence at all" the organization's members "intended to violate any state law or regulation or even that they will advocate such violations."

Webster added, "I have no doubt that the ancient halls of higher learning at Columbia will survive even the most offensive verbal assaults upon traditional moral values; solutions to tough problems are not found in the repression of ideas."

The decision applies only in the states comprising the 8th Circuit: Arkansas, Iowa, Minnesota, Missouri, Nebraska, North Dakota and South Dakota. By letting it stand, without comment, the Supreme Court majority avoided a choice, at least for the time being, whether it should apply everywhere or nowhere.

Rehnquist, in his dissent, wrote that first the trial court and then the appeals court "were doubtless as chary as we are of being thrust into the middle of this controversy but nonetheless obligated to decide the case."

But the Supreme Court's unquestioned discretion to decline to hear cases on the merits, "without explaining our reasons," should not "be used as a sort of judicial storm cellar to which we may flee from controversial or sensitive cases," Rehnquist said.

Joined by Justice Harry A. Blackmun, he went on to say that the court's rules do not "properly" allow it to "leave this important question of law" in its present unsettled state. Chief Justice Warren E. Burger also dissented.

So far as observers could recall, yesterday's action was the first in the Supreme Court to provide a victory, even if a limited and murky one, for the homosexual cause. Only five months ago, the court, without comment, handed that case a defeat by letting stand a ruling that the Tacoma, Wash., school board could fire James M. Gaylord for "immorality" --consisting of his admission, after 13 years on a high school faculty, that he was homosexual.

In other cases involving the First Amendment, the court took actions in which freedom of religion was an issue. SEGREGATION A RELIGION?

A school in Hialeah, Fla, operated by the New Testament Baptist Church barred two black children on the ground that its faith incorporates racial segregation as one of its tenets.

In July, the 5th U.S. Circuit Court of Appeals ruled, 7 to 6, that even though sincerely held, the tenet does not rise to the level of protection afforded by the First Amendment.

But only five of the seven judges in the majority held that the beliefs underlying the tenet were insufficiently "religious." The other two rated the beliefs qualitied, but said this defense was outweighed by "the compelling governmental interest," mainly under the Thirteenth Amendment, "insecuring the rights of blacks freely to contract with private schools" so as to eradicate "one of the badges of slavery."

Seven judges -- the dissenters plus one of the judges who provided the swing votes for the majority -- agreed that the plurality was discriminating among religions on the basis of how a sect derives or expresses its beliefs.

The lawyers for both the church and the black children -- for different reasons -- asked the Supreme Court to review the decision, which had a potential to affect the use of private academies to circumvent school desegregation. Yesterday, the court let the ruling stand. CATHOLIC LAY TEACHERS

The court granted a National Labor Relations Board petition to review a decision by the 7th U.S. Circuit Court of Appeals that the federal labor law does not apply to lay teachers in Catholic schools (in this case, in Chicago and in northeast Indiana who teach secular as well as religious subjects.