Liberals have relied on the federal courts for years to make social change.

And whether it was abortion, intolerable prison conditions, the rights of mental patients or racial discrimination, the federal bench didn't disappoint them.There seemed to be an endless supply of adventurous judges willing to experiment, to help them out.

Liberals will be looking to the courts in the next few years, more than ever, to counter the Reagan administration's thrusts against many of those same causes. Lawsuits challenging budget cuts and regulatory retrenchment al ready are tricking into the federal courts, and within a year they may become a flood.

But this time the results could be very different, thanks to the U.S. Supreme Court. The message from the high court to the low courts, as far as social experimenting is concerned, is becoming increasingly clear: Don't make waves. Don't wrinkle the sheets. Know your place and stay in it.

Beyond that, the court's rhetoric in some cases since the November election is sounding more and more like the Republican Party's 1980 platform statement.

Last week the court shot down federal judges in Pennsylvania who resorted to stringent and, for the state, costly orders to remedy "warehousing" of the retarded at a state-run institution, the Pennhurst State School and Hospital. Thomas K. Gilhool, one losing lawyer involved in the case, said the Supreme Court's decision "could have been written by David Stockman." The reference was to President Reagan's penny-pinching budget boss.

On the same day, the justices overturned the ruling of a Tennessee federal judge who had said the city of Memphis could not build a barrier between an all-white and an all-black neighborhood.

The month before, the U.S. Court of Appeals for the District of Columbia was put in its place when the justices, hailing free "market forces," said the appeals court had gone too far in requiring the Federal Communications Commission to review radio entertainment programming to protect against the loss of "unique" and minority-oriented shows.

Earlier that same month the court reversed the ruling of a federal judge in Illinois who had said mental patients were being discriminated against by the Medical program. Last year, two judges who ordered continued Medicaid abortion funding were similarly reversed. So was the Alabama U.S. district judge who restructed the city government of Mobile because he thought its at-large election system excluded blacks from public office.

The theme is not consistent and it is not new. But it is more pronounced now than ever, and smart lawyers -- like many in the American Civil Liberties Union -- are reworking their strategies to suit it, trying to avoid Supreme Court confrontations over many issues that matter to them in order to minimize their losses. It is better not to appeal a lower-court defeat to the Supreme Court, for there your cause can be totaled. Sometimes it is better to hunker down.

Behind the high court rulings the philosophy is hard to discern, because the justices have taken to speaking with many voices, through plurality opinions and multitudinous concurrences on many of the issues.

Collectively, however, the vision embodied in the opinions is of a humbler judiciary, occupying a modest place in the cosmos of government, deferring to other branches whenever possible, reading Congress narrowly, avoiding confrontations, taking it all a bit easier.

Those who approve of this course might say it shows greater respect for the separation of powers and for the majority will, as expressed through elected officials. Those who disapprove might call the court "chicken," or political, going with the perceived tide of conservative opinion.

Whichever, none of it bodes well for welfare rights organizations, Nader-style activists, labor unions, or others who will go to court against what the Reagan administration is doing.

The Pennhurst opinion, issued April 20, is perhaps the best recent case in point. In 1975, Congress enacted the Developmentally Disabled Assistance and Bill of Rights Act. The law was designed to improve the treatment of the retarded, the autistic, victims of epilepsy or cerebral palsy, among other illnesses, by dispensing money to the states and establishing guidelines for treatment.

The "Bill of Rights" section of the act stated in part that the afflicted "have a right to appropriate treatment, services and habilitation;" that these should be "designed to maximize the developmental potential of the person and should be provided in the setting that is least restrictive."

The U.S. district judge and the appeals court judges in the Pennhurst case confronted an institution that they regarded as "abominable" in its overcrowding, its violence, and its sanitary conditions.

For the appeals court, the act was clear enough: It created substantive right for the retarded, enforcable by federal alw. Those right were violated by Pennhurst, the court said as it ordered the institution all but cleared of residents. The courts, if effect, placed the institution in receivership and ordered the state to find better care for everyone.

It was judicial championship of a social cause, the sort of activism that stirred controversy in the past when judges took over prison systems and school systems. But the public-interest lawyers involved in the case believed the ruling might be safe from Supreme Court reversal because, unlike many others, the appeals court decision was based largely on an act of Congress, not on a judge's creative reading of the Constitution.

Yet even that was too much for the Supreme Court.Justice William Rehnquist, who wrote the opinion, was joined by Chief Justice Warren Burger and Justices Potter Stewart, Lewis Powell and John Paul Stevens in the majority view.

If Congress wanted to grant such sweeping authority, Rehnquist wrote, it would have said so. The Developmentally Disabled Bill of Rights was just an "encouragement" to the states, not an obligation, he said.

"We would be attributing far too much to Congress if we held that it required the states, at their own expense, to provide certain kinds of treatment," Rehnquist wrote.

So the Pennhurt ruling withdrew the court one step from the fray. If Congress wants the courts to enforce sweeping social reforms, it seemed to be saying, Congress will take the rap for it.

Rehnquist is, in fact, the foremost proponent on the court of a legal theory ressurrected from pre-New Deal days called the "non-delegation doctrine." Congress, this approach goes, should be done about controversial social issues to either the bureaucracy or the courts. When it does, Rehnquist said last year in a dissent in a dispute over benzene exposure regulations, the courts should nullify the results.

"It is the hard choices, not the filling in of the blanks, which must be made by the elected representatives of the people," Rehnqusit wrote in Industrial Union Department vs. Merican Petroleum Institute. "When fundamental policy decisons underlying important legislation about to be enacted are to be made, the bucks stops with Congress and the president . . ." e

The court's treatment of other kinds of social controversies also does not bode well for the Reagan administration's challengers.

When the justices ruled that statutory rape laws are constitutional -- after such laws had been attacked for treating men differently than women -- the Supreme Court stressed female chastity.

And when it upheld a Utah law requiring parental notice before preforming an abortion on a minor woman, it cited "family integrity."