The Supreme Court ruled 7 to 1 yesterday that the right of an individual Indian to the equal protection of the laws must yield to the right of the tribe to decide who its members will be.

The decision upholds the custom of the Santa Clara Pueblo of excluding children of a mixed marriage in which the woman is a Santa Claran. The tribe admits as members children of mixed marriages in which the man is a Santa Claran.

Julia Martinez, the woman who brought the suit, married a full-blooded Navajo in 1941. The couple had a 10 children. The eight survivors, all adults, live on the reservation in northern New Mexico. They have been accepted into the tribe's ancient religion, speak its unique language, Tewa, and pratice traditional customs.

Martinez has been trying for 32 years - until 1963 in the tribal government, after that in the federal courts - to compel the tribe to enroll her children.

The tribe dates back at least 600 and maybe 700 years. For most of that period, mixed marriages were rare. But by 1935, with isolation diminishing, such marriages occurred often enough to put strain on the tribe's limited resources.

In 1939, two years before Martinez married the tribe gave up a four-year practice of dealing case-by-case with the membership of children of mixed marriages. Instead, it passed an ordinance automatically granting membership to mixed-marriage children when their father is a Santa Claran and denying it when the mother is.

Deprived of membership, Martinez's children can't vote in tribal elections or hold secular office in the pueblo. Upon their mother's death, they will be expelled from the reservation and will be unable to inherit her home or her interest in the tribe's commonly owned lands.

For the federal courts, troubling problems were posed by the relevant law, the Indian Civil Rights Act of 1968, because it has what Justice Thurgood Marshall, in the opinion for the court, termed "two distinct and competing purposes."

One is to strengthen the position of individual tribe members by prohibiting a tribe from denying them the equal protection of the laws. The other is to furthe tribal self-government.

Although the court was "reluctant" to alter the balance between the two goals, Marshall wrote, tribal sovereignty must prevail because, in the absence "of any unequivocal expression of contrary legislative intent, . . . suits against the tribe under the ICRA are barred by its sovereign immunity from suit."

The dissenter, Justice Byron R. white, cited the concern Congress showed, in the period of more than three years preceding passage of the 1968 law, about deprivations of Indian rights by tribal authorities.

"I cannot believe, as does the majority," he wrote, that Congress "desired the enforcement of these rights to be left up to the very tribal authorities alleged to have violated them.

"In the case of the Santa Clara," he continued, "both legislative and judicial powers are vested in the same body, the Pueblo Council. To suggest that this tribal body is the 'appropriate' forum for the ICRA is to ignore both reality and Congress' desire to provide a means of redress to Indians aggrieved by their tribal leaders."

Justice Harry A. Blackmum, who was ill when the case was argued, did not participate.

The court took other actions: ALLIENS

In March departing from an established trend, the court ruled that New York State as not denying the equal protection of the laws guaranteed by the Constitution with a requirement that state troopers be citizens. In three aliens cases yesterday, the court:

Sent back to a lower court, "for further consideration in light of" the March decision, a ruling invalidating a California requirment that probably officers be citizens.

Agreed to review a decision invalidating a New York exclusion of aliens who haven't applied for citizenship from eligiblity for permanent teaching certificates.

Let stand a decision invalidating a New York requirment that aliens licensed as physicians must become citizens for 10 years face cancellation of their licenses. SODOMY LAWS

With Justices Brennan and Marshall dissenting, the court let stand a North Carolina law that makes it a crime for consenting adults, in private, to enage in homosexual acts. Such laws are on the books of most states. The case involved the conviction of Eugene Enslin of Jacksonville, N.C. FOREIGN SERVICE RETIREMENT

The court agreed to review a decision that mandatory retirement at age 60 for about 8,000 diplomats and others covered by the Foreign Service retirement system denies the equal protection of the laws because Civil Service employes, including those serving overseas, didn't have to retire until 70. PANAMA CANAL

Balking the latest in a series of challenges from Capitol Hill to the Panama Canal pacts, the court let stand a ruling that the President and the Senate, acting by a self-excuting treaty, may transfer to another nation property in which the United States has an interest.