The Supreme Court declined yesterday to give mentally retarded people extra legal protection against discrimination. But it nevertheless said Texas officials acted improperly in barring a home for the retarded in Cleburne, Tex.

The court, which expected to conclude the 1984-85 term today, voted, 6 to 3, to overturn an appeals court ruling that judges should give "heightened scrutiny" to laws that might unfairly affect the retarded.

Such scrutiny, which is used by judges under the Constitution's equal protection clause, has been afforded to laws that might discriminate against women, for example, but not to laws involving age discrimination.

Justice Byron R. White, writing for the majority, said the appeals court was wrong in requiring a "more exacting standard of judicial review." White said that "those who are mentally retarded have a reduced ability to cope with and function in the everyday world" and range in disability from those with only minor problems to "those who must be constantly cared for."

"They are thus different," White said, and the "states' legitimate interest in dealing with and providing for them is plainly a legitimate one."

But White struck down the Cleburne ordinance, which required group homes for the mentally retarded to obtain special permits before they could open in residential neighborhoods but did not require such permits for nursing homes, schools or apartment buildings.

White explained that, in general, laws that might affect certain groups more than others will be upheld if the "classification drawn by the statute is rationally related to a legitimate state interest."

Statutes involving classifications based on race, alienage and national origin must meet a "compelling state interest" and be carefully checked by judges.

Classifications based on gender must be "substantially related to a sufficiently important governmental interest," and they, too, require a "heightened scrutiny" by judges. It is that standard that the appeals court suggested would be appropriate for considering laws that affected the mentally retarded.

The high court yesterday refused to place mentally retarded people in that same category.

But White said the Cleburne officials acted unconstitutionally because "requiring the permit in this case appears to us to rest on an irrational prejudice against the mentally retarded."

Chief Justice Warren E. Burger and Justices Lewis F. Powell Jr., William H. Rehnquist, Sandra Day O'Connor and John Paul Stevens also joined White's opinion in City of Cleburne v. Cleburne Living Center.

Stevens, joined by Burger, said the appeals court "correctly observed that through ignorance and prejudice the . . . retarded 'have been subjected to a history of unfair and often grotesque treatment.' "

Stevens said the Cleburne zoning permit was required as a result of "the irrational fears of neighboring property owners, rather than for the protection of the mentally retarded . . . ."

Justices Thurgood Marshall, William J. Brennan and Harry A. Blackmun dissented.

Marshall said the mentally retarded were subjected to a "segregation and degradation . . . that in its virulence and bigotry rivaled and indeed paralleled the worst excesses of Jim Crow."

The majority was taking a "novel" approach to handling constitutional claims involving equal protection, he said. "No guidance is thereby given as to when the court's freewheeling, and potentially dangerous, 'rational basis standard' is to be employed," Marshall said.

American Civil Liberties Union litigation director Burt Neuborne said the court's ruling is nonetheless a victory. "Now you don't need to be in a 'suspect class' . . . to win an equal protection case," Neuborne said.

Most observers had assumed that the government would generally win against claims of equal protection violations unless plaintiffs could show that they are in one of a few protected classes.

"The opinion establishes a 'rational-basis' test as an effective protection," Neuborne said. The government, under that test, "will have to come forward and prove that it is acting on something more than stereotypes and prejudice."

In another case decided yesterday, the court gave border officials more power to stop and detain for lengthy periods anyone they suspect might have swallowed drugs to bring them into this country.

The case, U.S. v. Rosa Elvira Montoya de Hernandez, involved what officials called an "alimentary canal" or "balloon swallower" -- someone who tries to bring in drugs by swallowing them while outside this country and then excreting them, once across the border.

In this instance, a woman flying into Los Angeles from Bogota, Colombia, was detained after Customs officials noted that she had made several recent, short trips to this country, had paid cash for her ticket, was carring $5,000 in cash and had no relatives or place to stay.

Border officials said she could return to Colombia on the next flight, consent to an X-ray or remain in detention until "she produced a monitored bowel movement," Rehnquist said. She chose to return to Colombia but no flight could be found.

She remained in detention for 16 hours, refusing food and drink or to use the restroom until officials got a court order for a rectal examination.

Over the next four days, the woman excreted 88 balloons totaling about one pound of cocaine. She was convicted of drug smuggling, but the 9th U.S. Circuit Court of Appeals overturned her conviction.

The high court, in an opinion by Rehnquist, voted, 7 to 2, to uphold her conviction. Rehnquist said the detention is justified when agents "reasonably suspect" that someone is bringing drugs into the country.

"Her visible efforts to resist the call of nature, which [the appeals court] labeled 'heroic,' " caused her humiliation and discomfort," Rehnquist said.

Brennan, joined by Marshall, dissented, calling the detention and search a "disgusting and saddening episode at our nation's border."