The Supreme Court, siding with the Reagan administration in a crucial case, yesterday sharply curtailed the ability of citizens' groups to use the federal courts as a way of enforcing the separation of church and state.

The court said in a 5-to-4 decision that Americans United for Separation of Church and State, a prominent nationwide organization devoted to challenging government benefits for religion, had no right to sue in federal court to block transfer of a surplus military hospital to a Christian college in Pennsylvania. The court said the transfer had not hurt the group's 90,000 members directly.

In making its ruling, the court issued one of its strongest denunciations of judicial activism, saying that the courts were not "constituted as ombudsmen of the general welfare," "college debating forums" or podia for those who would "roam the country in search of governmental wrongdoing."

Yesterday's case was watched closely by the administration as part of its campaign for "judicial restraint." Keeping such groups out of the courts is central to the administration's effort to limit the judiciary's power to intervene with the other branches of government and with the states.

It also was watched as a test of Justice Sandra D. O'Connor's commitment to Reagan conservatism, and she did not fail the president yesterday. O'Connor voted with the majority, joining an opinion written by Justice William H. Rehnquist.

Lawyers interviewed yesterday said the decision clearly would create a new obstacle for those contesting church-state issues, such as the use of government property for religious exercises.

Beyond that, many suggested that the ruling would send a clear message to the lower courts to shut the door on many kinds of "public-interest" suits brought under the Constitution.

Solicitor General Rex E. Lee called it the most significant ruling in a decade concerning the right to sue. "It will substantially affect the extent to which citizens can raise public-interest issues in court," he said.

Americans United sued after the 1976 transfer of the surplus property to the Valley Forge Christian College near Philadelphia. Ordinarily, anyone suing must show that the challenged action hurt them in some concrete way: a loss of money or property. That requirement has, in the past, been relatively relaxed in religion cases.

When it sued, Americans United claimed that its members suffered from the transfer because they were taxpayers and lost money by the donation and because they were citizens whose right to enforcement of the Constitution had been denied. The Court of Appeals for the Third Circuit allowed them to proceed as citizens.

Yesterday the court reversed that decision, saying that neither rationale was acceptable and that to accept either would upset the balance of power between the courts and the rest of government.

Rehnquist said that "any connection between the challenged property transfer and the group's tax burden is at best speculative and at worst nonexistent." The money to build the hospital was spent 30 years ago, he said, and the state government determined that the property was useless to it, though it was valued at more than $1 million.

Nor had the group suffered a concrete injury because of an alleged violation of the Constitution, he said, "other than the psychological consequence presumably produced by observation of conduct with which one disagrees."

"Their claim that the government has violated the Establishment Clause does not provide a special license to roam the country in search of governmental wrongdoing and to reveal their discoveries in federal courts," he said. "The federal courts were simply not constituted as ombudsmen of the general welfare."

Dissenting in Valley Forge Christian College vs. Americans United for Separation of Church and State were Justices William J. Brennan Jr., Thurgood Marshall and Harry A. Blackmun. Justice John Paul Stevens wrote a separate dissent.

They said that cases brought under the Constitution's prohibition of establishment of religion should receive special treatment by the courts.

In other action yesterday:

In a case involving federal pay scales, the court ruled unanimously that government employes transferred from blue-collar to white-collar status get a lesser pay increase than they had sought. In U.S. vs. Clark, the blue-collar employes, part of the government's "prevailing wage" pay system, had argued that such a transfer automatically entitles them to a two-step salary increase. Justice O'Connor wrote that their salary treatment, instead, must be determined by the highest previous salary they had received.

In a 5-to-4 ruling, the court upheld an Indiana law that terminated mineral rights beneath a parcel of land for owners who fail to exercise them for 20 years. The law was designed to free land of legal questions that could tie up sale or development. Those challenging the law in Texaco Inc. vs. Short said it stripped them of property without compensation.