The Supreme Court yesterday erected major new hurdles for criminals challenging their convictions in the nation's federal courts.

Justice Sandra Day O'Connor, writing for the court in four separate habeas corpus cases, said the obstacles were necessary to assure "finality" of punishment in the criminal justice system and to free state judicial systems from excessive scrutiny by federal judges.

Three of yesterday's defendants went to the federal courts seeking their freedom because of flaws in their trials in the state courts of Ohio. They had not made these objections at the time of their trials, however, and O'Connor said they therefore lost their right to raise them in the federal court.

A fourth defendant, Joseph C. Frady, was convicted of a brutal 1963 murder in the District of Columbia. He subsequently spent nearly two decades filing petitions until he won his freedom in February, 1981. He now will be returned to jail.

These cases, while not individually overwhelming in their impact, are part of a recent cumulative attempt by the court to limit the use of federal courts by prisoners. They are also central to a long-running law-and-order debate in which conservatives, including Reagan administration officials, are attempting to reduce what they call the "endless appeals" allowed by earlier Warren Court rulings and to restore judicial authority to state criminal courts.

The president appointed O'Connor to the court in part because of her strong views, as a state appellate judge in Arizona, on this controversy.

All convicted criminals are guaranteed the right to appeal their convictions. Federal law, and the Constitution, also permit them to challenge their convictions even after they have lost their appeal through use of federal habeas corpus petitions. The fact that relatively few petitions are successful has not stifled the debate.

The Ohio defendants had appealed convictions for various violent crimes and had lost. They then began filing habeas corpus petitions challenging the jury instructions used by the judge at their respective trials. None had made these objections at their trial.

They won in the lower federal courts, claiming that Ohio law on jury instructions had changed between their trials and their petitions. O'Connor said that except where there was a miscarriage of justice, the defendants had no right to raise the new objections.

"The writ of habeas corpus indisputably holds an honored position" in American law and the Constitution, O'Connor wrote in the 6-to-3 ruling. But it "entails significant costs," extending "the ordeal of trial for both society and the accused" and undermining "the usual principles of finality of litigation."

In addition, successful habeas corpus petitions years after a crime often free a defendant long after witnesses are dead or their memories eroded, she said. Thus they "may reward the accused with complete freedom from prosecution."

"Finally, the Great Writ imposes special costs on our federal system. The states possess primary authority for defining and enforcing the criminal law . . . . Federal intrusions into state criminal trials frustrate both the states' sovereign power to punish offenders and their good faith attempts to honor constitutional rights."

Justice Harry A. Blackmun wrote a separate concurrence in Engle v. Isaac, the case filed by the three Ohio defendants. Justice John Paul Stevens also wrote separately, agreeing in part and dissenting in part. Justice William J. Brennan Jr. was joined by Justice Thurgood Marshall in a vigorous dissent.

They accused O'Connor of distorting facts in the case to reach her result, a "conspicuous exercise in judicial activism," they said, and of disregarding her own ruling a few weeks ago in a case called Rose v. Lundy.

"Sic transit gloria Lundy," Brennan punned. "In scarcely a month, the bloom is off the Rose."

In other action yesterday:

* Justice William H. Rehnquist, writing for the majority, struck down a Texas law that restricted the right of illegitimate children to file paternity suits in order to obtain financial support from the father.

The law placed a one-year statute of limitations on paternity suits filed on behalf of illegitimates. That meant that once past the first birthday, the child had no recourse.

Rehnquist said that since legitimate children have a greater opportunity to seek paternal support, the Texas limitation denied equal protection to illegitimates.

The vote in Mills v. Hableutzel was unanimous although four justices expressed separate views.

* The court ruled unanimously that the Kentucky courts had interpreted too broadly the bribery laws when they said they included a candidate's campaign promise to reduce his salary if elected.

The candidate for Jefferson County commissioner had been denied his election victory by state courts after they ruled that his promise constituted an offer of money to voters in exchange for votes, that is, vote-buying.

Brennan, writing for the court in Brown v. Hartlage, said the application of the state's vote-buying law violated the First Amendment's free speech guarantee.