The Supreme Court, reaffirming that it would carefully scrutinize any form of sex discrimination, ruled 5 to 4 yesterday that the nursing school at the nation's oldest publicly supported all-female college may no longer exclude men.

Justice Sandra Day O'Connor wrote the opinion in a notable departure from her state's rights campaign and from some of her usual allies, who mourned the death of a tradition in their dissent.

O'Connor said the exclusion of men from the Mississippi University for Women nursing school served only to "perpetuate" stereotypes that some jobs are for women and some are for men.

And in language called crucial by women's rights lawyers, O'Connor underscored what she called the "firmly established" principles that "do not vary," and which the court should bring to any cases involving legal distinctions between men and women.

Those principles, requiring "exceedingly persuasive" justification for any gender distinction in the law, appeared to be softening considerably at the court last year.

Though the actual judgment yesterday apparently applies only to one school and directly benefited a man, women's rights advocates were anxiously watching it as the Equal Rights Amendment deadline passed for clues as to how the court will treat future sex discrimination cases.

"This is a key decision," said Phyllis Segal, a prominent women's rights attorney. "They were pointed in a very different direction last year. We were very concerned."

Eleanor Smeal, president of the National Organization for Women, said the ruling was particularly welcome "on the day after the ERA deadline," but expressed concern about the closeness of the decision.

Chief Justice Warren E. Burger, and Justices Harry A. Blackmun, Lewis F. Powell Jr., and William H. Rehnquist dissented. Blackmun and Burger wrote separate dissents.

"I have come to suspect that it is easy to go too far with rigid rules in this area of claimed sex discrimination," said Blackmun.

"The court's opinion bows deeply to conformity," wrote Powell, joined by Rehnquist. "Left without honor--indeed, held unconstitutional--is an element of diversity that has characterized much of American education and enriched much of American life."

The ruling, based on the 14th Amendment's requirement that governments dispense equal treatment under the laws, does not apply to private colleges, although it will likely be used to file suits against private schools.

The Mississippi University for Women, in Columbus, is one of the country's only sex segregated public institution of higher learning. Established in 1884, it is also one of the oldest.

Joe Hogan, who brought the suit, is a licensed practical nurse and resident of Columbus. He attempted to enroll in the school's nursing program in August, 1976, to obtain a college degree in nursing. He was denied admission because he is male. Hogan reapplied unsuccessfully in 1979 before filing suit in U.S. District Court.

Hogan lost at the district court level. But the 5th U.S. Circuit Court of Appeals held that the all-female college violated the Constitution's equal protection provision and amounted to an unjustified act of sex discrimination.

Numerous forms of sex discrimination, including differences in liquor laws, Social Security benefits and terms of employment have been struck down in the last decade under the court's test, which allows gender distinctions only when they are "substantially related to an important governmental objective."

Last year, in rulings upholding statutory rape laws punishing only men and the exclusion of women from the military draft, the court seemed to be abandoning these requirements or changing them from case to case.

O'Connor wrote yesterday that those requirements "do not vary simply because the objective appears acceptable to individual members of the court."

She rejected Mississippi's explanation for excluding men, that an all-female nursing school was a form of affirmative action for women, by noting that women dominate the nursing field and hardly need any help getting in.

In fact, "rather than compensate for discriminatory barriers faced by women, MUW's policy of excluding males from admission to the school of nursing tends to perpetuate the stereotyped view of nursing as an exclusively woman's job. By assuring that Mississippi allots more openings in its state-supported nursing schools to women than it does to men, MUW's admissions policy lends credibility to the old view that women, not men, should become nurses, and makes the assumption that nursing is a field for women a self-fulfilling prophecy."

In another ruling yesterday, the court unanimously upheld a California law allowing liquor distillers to choose a single distributor in the state. The court said the law, similar to those in a number of other states, does not automatically violate federal antitrust laws. Rehnquist wrote the opinion in Rice vs. Norman Williams Co., et al.

In a third case, the court said Puerto Rico may sue Virginia apple growers on behalf of Puerto Rican workers who were allegedly unfairly treated in the allocation of seasonal apple orchard jobs. Justice Byron R. White, in a unanimous ruling in Alfred L. Snapp & Son, Inc., et al. vs. Puerto Rico, said the state's interest in the well-being of its residents entitled it to "parens patriae" status, meaning literally "parent of the country."