The U.S. Supreme Court, emphasizing the need to limit repeated challenges to criminal convictions, yesterday reinstated a 1963 murder conviction that had been thrown out by the federal appeals court in Washington after the defendant had spent 17 years in jail, two of them on death row.

The high court's decision, written by Justice Sandra Day O'Connor, came in the case of Joseph C. Frady, who with a codefendant was convicted of first-degree murder in 1963 and sentenced to die in the electric chair for the killing of a Southeast Washington man who was bludgeoned to death with a table leg.

Frady, whose death sentence was commuted to life in prison, was freed in February 1981 after the U.S. Court of Appeals said the trial judge in his case had given the jury instructions that were found to be improper in another case decided five years later.

O'Connor noted in her opinion that the argument that finally won Frady his freedom--raised for him by a jailhouse lawyer in 1978--marked the ninth time since 1965 that Frady had attacked his conviction.

Frady, who is living with a daughter in rural Virginia, could not be reached yesterday. If he had not been released from jail, he would have been eligible for parole in 1983. The high court decision yesterday apparently means that Frady must return to jail.

When the appeals court panel reversed Frady's conviction in 1980, in a decision written by Judge Harry T. Edwards, it said that the improper instructions given by Judge George L. Hart Jr. amounted to "plain error" that seriously affected Frady's rights and thus warranted a reversal. A majority of the full court, over strong dissent from four members, refused the government's request to rehear the case.

O'Connor, joined in her opinion by four other justices, yesterday said that the appeals court was wrong when it applied the "plain error" standard.

That standard, O'Connor said, was meant to be used only on appeal brought directly after conviction, when the court is compelled to promptly correct "miscarriages of justice" in a matter considered vital to the defendant's rights.

Once a prisoner's chance to appeal is exhausted at that stage, however, O'Connor said, "we are entitled to presume he stands fairly and finally convicted. . ." When the appeals court in 1980 applied the "plain error" standard to Frady's long-delayed challenge to the jury instructions, it was "as though the clock had been turned back to 1965" and effectively gave Frady the chance for a second appeal, O'Connor asserted.

When a prisoner enters a new challenge years after his conviction, O'Connor said, it is a "well-settled principle" in the courts that he faces a tougher standard if he wants to overturn his case. The defendant must demonstrate a specific cause that prevented him from raising that challenge during the original appeal and he must show that he suffered actual prejudice as a result of the error at his trial, O'Connor said.

In a case involving jury instructions, O'Connor said the courts have ruled that a defendant must show not just that the instruction was erroneous, but also that it so infected the entire trial that the conviction was unconstitutional.

Frady had argued that the judge's instructions to the jury assumed that the prosecution had proved that he acted maliciously and thus ruled out the possibility that he could have been found guilty of the lesser crime of manslaughter.

O'Connor agreed with the dissenting appeals court judges that the evidence against Frady showed "malice aplenty" and said that Frady failed to show that the judge's instruction actually put him at a disadvantage during his trial.

Calling the crime a murder of "unspeakable brutality," O'Connor said that the facts in the case, plus Frady's "utter failure" to show he acted without malice, disposed of any claim of prejudice that would justify reversing his conviction 19 years after the fact.

"We perceive no risk of a fundamental miscarriage of justice in this case," O'Connor said. She was joined by Justices Byron R. White, Lewis Powell, William Rehnquist and John Paul Stevens.

Justice Harry Blackmun joined in the outcome of the case, but writing separately said he felt that the "plain error" rule could be applied--but would not be met--in Frady's case. Justice William Brennan dissented, also saying the plain error rule should apply. He added, however, that "given the facts of this case," it would have been hard for him to uphold the reversal of Frady's conviction.

Chief Justice Warren Burger, who was on the Washington appeals court at the time the Frady case was heard, and Justice Thurgood Marshall, who was solicitor general at the time, did not participate in the decision.