A divided Supreme Court yesterday refused to rule on whether judges have the right to penalize unmarried couples living together, thereby rejecting one of the most controversial cases to come before it this year.

The court, with three justices dissenting publicly, let stand an Illinois state court ruling that stripped Jacqueline Jarrett of custody of her children because she lived with her boyfriend. The lower court ruling said that the living arrangement jeopardized the "moral and spiritual well-being of the children."

At the same time, the Supreme Court let stand an appellate ruling that the U.S. government can ban the use of Laetrile, whose advocates claim it is a cancer cure.

The Jarrett child custody case action-fueled an already heated debate about laws against unmarried people living and sleeping together and about the extent to which judges should impose their own morality on others.

The dissenting justice joined the debate yesterday. "Nothing in the record or in logic," said Justice William Brennan with Justice Thurgood Marshall, "supports a conclusion that divorced parents who fornicate, for that reason alone, are unfit or adversely affect the well-being and development of their children . . ."

Pointing out that there are at least 1.1 million couples living like Jarrett and her boyfriend, Brennan and Marshall said the issue should be addressed because it "has important implications for many households." Justice Harry Blackmun also dissented from the court's action, leaving the case just one vote shy of the four needed for review.

It was, in fact, the children's father, Walter Jarrett, who raised the moral objection. After their 1976 divorce, Jacqueline was given custody of the couple's daughters, then aged 12, 10 and 7. Walter Jarrett sought custody four months later, after learning that Jacqueline was about to start living with another man. Walter Jarrett successfully argued that his daughters would be raised in an "immoral atmosphere."

The Illinois courts sided with him, citing a state law against unmarried couples living together. That made Jacqueline a lawbreaker, the court ruled, and unfit for custody. The Illinois courts did not inquire into the actual impact on the children of their mother's living arrangement.

Brennan and Marshall called the reasoning "irrationally overbroad. It would make no sense to treat murder, fornication and traffic violations similarly for purposes of custody modification," they said. "If Jacqueline had violated Illinois' traffic laws, she might have lost her driver's license, but surely not custody of her children."

In other action yesterday, the court:

Declined to rule on laws banning the use of Laetrile, the controversial drug some claim cures cancer. The justices let stand a federal appeals court ruling that the government is free to ban Laetrile use in the United States. The Food and Drug Administration has never allowed Laetrile to be sold in interstate commerce on the grounds that it has not been demonstrated to be effective in treatment of terminal cancer.

Agreed to consider whether a military pension is "community property" to be divided between a husband and wife in the event of a divorce. The question has arisen with increasing frequency in recent years. Seven state courts have ruled that one spouse's military pension, like furniture or other property, belongs also to the other spouse. Four state courts have diagreed. The case accepted for argument yesterday comes from California, where the state Supreme Court ruled that the military retirment benefits earned by Richard J. McCarthy for 20 years of service should be divided with his ex-wife, Patricia Ann McCathy. The couple was divorced in 1976.

Agreed to consider the limits of legitimate federal civil rights suits brought by inmates against prison officials. The case (Parratt vs. Taylor) stems from a $23.57 civil rights award to a Nebraska inmate who sued because a mail order hobby kit got lost after arriving at the prison. There is relatively little debate about the use of civil rights suits to correct alleged brutality and major civil rights violations in prisons. But wardens have been subjected to an increasing barrage of lesser suits by inmates increasingly knowledgeable about law.

Accepted for review a water pollution suit brought by commerical clammers in 1977 charging that New York, New Jersey and the federal government, by dumping sewage into the Atlantic or allowing it to be dumped, were ruining their livelihoods.

The issue in four separate cases brought by the National Seal Clammers Association is whether they, as opposed to the government, are allowed to bring such a suit under the Federal Water Pollution Control Act.