The Supreme Court, treading on one of the major battlefields of state-federal relations, ruled yesterday that the states cannot be forced to spend large sums of money conforming to vague federal laws governing treatment of the retarded.

The 6-3 ruling is expected to impede nationwide efforts to remove the retarded and other "developmentally disabled" people from warehouse-style institutions and place them in a community setting.

It also serves as a warning being repeated with increasing frequency by the Supreme Court: Lower federal judges should not read into acts of Congress sweeping social reforms for the states. If Congress wants such broad remedies, Justice William Rehnquist wrote, it will have to specifically legislate them.

The justices reversed a lower court order requiring the state of Pennsylvania to clean out the Pennhurst State School and Hospital. A federal district judge in 1977 described Pennhurst as "abominable," "unsanitary," overcrowded, violent and "actually hazardous to residents" before making the state consider appropriate and potentially costly alternatives for each of Pennhurst's 1,230 retarded residents.

The 3rd U.S. Circuit Court of Appeals upheld most of the District Court's actions, basing its decision on the Developmentally Disabled Assistance and Bill of Rights Act of 1975. That law, like dozens of others enacted during the past two decades, gave federal money to states to improve conditions for the afflicted on the condition that the states meet certain requirements. But, as in many of those other laws, Congress was not specific in defining the requirements.

The lower courts, at the request of the U.S. government, various organizations representing the retarded and some of the Pennhurst residents, made their own determination of what Congress meant. The controversy, joined by other states fearing costly court orders, soon took on the theme of getting the federal government "off the backs of the states."

Rehnquist wrote yesterday that these laws are essentially "contracts" between the federal government and individual states. If the conditions imposed by the contracts on the states are unclear, the states cannot be expected to meet them.

"Accordingly, if Congress intends to impose a condition on the grant of federal moneys, it must do so unambiguously," he wrote. "By insisting that Congress speak with a clear voice, we enable the States to exercise their choice knowingly, cognizant of the consequences of their participation."

Turning to the specific law in question, Rehnquist said the "Bill of Rights" section, hailed at the time as a major step forward for the retarded, the autistic, victims of cerebral palsy and other disabilities, was vague. It said such persons have a right to "appropriate" treatment designed to maximize their potential in the "least restrictive" environment. But it did not say how the states were supposed to do this.

It was "too thin a reed to support the rights and obligations read into it by the court below," Rehnquist said. Nothing "reveals an intent [by Congress] to require the states to fund new, substantive rights. . . . We would be attributing far too much to Congress if we held that it required the states, at their own expense, to provide certain kinds of treatment," he said.

The defeat for those favoring federally imposed changes was limited in yesterday's decision by the court's unwillingness to address several other issues in the case. It did not consider whether individual residents have the right to bring suit under the act. The federal government clearly has such power, but if it is not going to use it, the question remains, who can? That question is especially important in light of the Reagan administration's promises to decrease federal intervention in state affairs.

The court also declined to rule on whether the developmentally disabled have a constitutional right to improved conditions, independent of any specific law.

Justices Byron White, William J. Brennan and Thurgood Marshall dissented in the Pennhurst State School Vs. Halderman decision. Justice Harry Blackmun concurred in the judgment but disagreed with parts of the opinion.

In other action yesterday:

The court ruled unanimously that unions contributing through collective bargaining to sex discrimination at a company do not have to contribute if the company has to pay damages for the discrimination.

Northwest Airlines sought union help in paying a $20 million award won by women cabin attendants. The attendants won their case on the grounds that they were paid less than male cabin attendants.

But the pay differential was in part the result of a collective bargaining agreement with the Transport Workers Union of America (AFL-CIO) and when it came time to pay, Northwest went to court to get money from the union. w

Justice John Paul Stevens said yesterday that nothing in the law supported Northwest's contention that the unions also were liable. Justice Harry Blackman took no part in the case.

The court let stand a Kentucky ruling forcing a mother living in a lesbian relationship to give up custody of her 6-year-old daughter. The Kentucky appeals court said the removal of the daughter was "in the best interests of the child." The divorced father sought the change in custody, saying the relationship was a threat to the child's well-being. Last year, the Supreme Court left untouched an Illinois ruling removing custody from a mother living unmarried with her male lover.