Supreme Court Justice William H. Rehnquist said Thursday that the courts should be unavailable to individuals seeking to vindicate claims for redress of some wrongs involving such nongovernmental institutions as the family.

"There are times when the claims of the individuals should be subordinated to those of the "species" even if the species not be the government itself but a private institution which serves a useful purpose." Rehnquist said in a lecture at the University of Miami law school in Coral Gables. Fla.

The justice's principal case in point was "any sort of adversary hearing which pits parent against child," because it "is bound to be disruptive, placing stresses and tensions on the inta-familial relationships which in turn weaken the family as an institution."

He told of a couple in Stillwater. Minn., who, after the birth of their seventh child, consulted a physician about how to ensure that the family would grow no larger.

The doctor recommended, and the father had, a vasectomy. But a little more than a year after the operation, the mother delivered a healthy baby boy.

The parents sued the physician and his clinic for negligence.

A divided Minnesota Supreme Court upheld the parents' right to recover damages by proving their child to be a net burden.

In so ruling, however, the majority was troubled. The judges expressed "hope that future parents and attorneys would give serious reflection to the silent interests of the child and, in particular, the parent-child relationships that must be sustained long after legal controversies have been laid to rest."

Two dissenters termed the majority holding "contrary to public policy" and added. "We should not permit the courts to be used for this purpose."

Rehnquist said he "would be hard pressed to find any case which more clearly than this one embodies the adversary system of justice carried to its ultimate conclussion with all of the advantages and disadvantages that flow therefrom.

"What is the effect of such a lawsuit going to be upon the relationship between these parents and this child from the time of his birth through the subsequent 17 or 18 year which he will in the normal course of events spend with his parents?" Rehnquist wondered.

"To pit children against their parents in the court in order to determine whether an operation recommended by a physician should or should not be autorized by a court, may make for a better informed decision as to the propriety of the operation, but may leave the family unit in a shamble," the justies said.

More broadly, Rehnquist voiced "a great deal of doubt" about the adversary system for resolutions of centain other social problems and disputes.

For example, Rehnquist said, the Supreme Court has already interpreted the Constitution to bar a lay court from resolving a dispute betweem factions of a church. Similarly, he said, more than one court opinion contains "rather clear implication" that labor unions as an institution, along with collective bargaining, are subject to less "adversary exploration" in the court than are other institutions ineconomic life.