The Supreme Court, ruling yesterday for the first time on the use of hypnosis in the courtroom, said states may not prohibit testimony by criminal defendants whose memories have been enhanced through hypnosis.

The 5-to-4 decision struck down an Arkansas rule, similar to rules in 17 states, including Maryland, that flatly barred use of hypnotically assisted testimony on the ground that it tended to be unreliable.

Justice Harry A. Blackmun, writing for the court, said a defendant's constitutional right to testify in his own defense outweighed the state's interest in excluding possibly unreliable evidence.

Completely excluding such testimony, Blackmun said, "is an arbitrary restriction on the right to testify in the absence of clear evidence by the state repudiating the validity of all posthypnosis recollections."

The ruling, narrowly confined to criminal defendants, did not address more typical situations such as the use of hypnotically obtained testimony by witnesses who have no such constitutional right to testify.

In other cases yesterday, the court:

Banned 6 to 3 any mandatory death penalties for prisoners serving life sentences who kill guards or fellow inmates. Blackmun, writing for the court in Sumner v. Shuman, said such mandatory sentences violated defendants' rights under the Eighth Amendment's ban on cruel and unusual punishment.

Refused to disturb the fraud conviction of a Florida contractor despite allegations that a majority of the jurors at his trial were high on alcohol, marijuana and cocaine during the trial. Justice Sandra Day O'Connor, writing for the court in Tanner v. U.S., rejected the contractor's appeal for a hearing on the allegations.

Investigations into "allegations of juror misconduct, incompetency or inattentiveness raised {after the verdict} seriously disrupt the finality of the process," O'Connor said in the 5-to-4 decision.

Upheld 5 to 4 the death sentence of John H. Adamson for the 1976 murder of Arizona Republic reporter Donald Bolles. After agreeing to testify against others involved, Adamson pleaded guilty to a lesser charge and was sentenced to 20 years in prison. The other defendants had to be retried and Adamson refused to testify a second time. He was then convicted of murder and sentenced to death. Adamson claimed his constitutional right against double jeopardy had been violated and a federal appeals court agreed.

Justice Byron R. White, writing for the court in Ricketts v. Adamson, overruled the lower court, saying there was no constitutional violation because Adamson breached the terms of his plea agreement. Agreed to review next term an important affirmative action controversy over what employes must show in order to prove discrimination claims in promotions. The case, Watson v. Fort Worth Bank & Trust, is expected to be heard in early 1988.

The hypnosis case began four years ago, when Vickie Lorene Rock pulled out a gun and shot her husband to death after he tried to prevent her from leaving their apartment to get a hamburger.

Rock, who first could not remember many of the details leading up to the shooting, was hypnotized to help her recall the events. After hypnosis, she remembered that she did not have her finger on the trigger of the gun and that it discharged accidentally when her husband grabbed her arm.

As a result of her new recollection, a gun expert was called into examine the gun and determined that the gun was defective and prone to fire when hit or dropped.

The trial judge, upon learning of the hypnosis, barred testimony based on the hypnosis. The Arkansas state supreme court agreed, saying that such testimony could never be used.

Blackmun, joined by Justices William J. Brennan Jr., Thurgood Marshall, Lewis F. Powell Jr. and John Paul Stevens, overturned the state court but emphasized that the court was doing so because the ban involved a criminal defendant and because the state rule would not allow for exceptions.

"We are not now prepared to endorse without qualifications the use of hypnosis as an investigative tool," Blackmun said. "Scientific understanding of the phenomenon and of the means to control the effects of hypnosis is still in its infancy."

But "despite the unreliability that hypnosis concededly may introduce," Blackmun said, "the procedure has been credited as instrumental in obtaining investigative leads or identifications that were later confirmed by independent evidence."

"A state's legitimate interest in barring unreliable evidence," he said, "does not extend to {blanket} exclusions that may be reliable in an individual case."

Such testimony can be checked for reliability, he said, and states could establish "guidelines to aid trial courts in the evaluation of posthypnosis testimony and it may be able to show that testimony in a particular case is so unreliable that exclusion is justified."

But Arkansas "has not shown," Blackmun concluded, "that hypnotically enhanced testimony is always so untrustworthy and so immune to the traditional means of evaluating credibility {of witnesses} that it would disable a defendant from presenting her version of the events for which she is on trial."

Chief Justice William H. Rehnquist, writing in dissent, said the states should be left to determine their own rules in this area. "Until there is a much more general consensus on the use of hypnosis than there is now," Rehnquist said, "the Constitution does not warrant this court's mandating its own view of how to deal with the issue."

The mandatory death sentence case involved a question the court had left open since 1976 when it first began striking down similar death penalty laws.

The court consistently has said that death was different from any other form of punishment and that defendants must be given every opportunity to present mitigating evidence on their behalf.

That view prevailed in yesterday's case, which involved convicted Nevada murderer Raymond Wallace Shuman. Convicted of first-degree murder 29 years ago, Shuman was serving life without parole when he killed an inmate in 1975 by dousing him with gasoline and setting him on fire.

Blackmun, joined by Brennan, Marshall, Powell, Stevens and O'Connor, rejected arguments that the mandatory death sentence was needed to deter "lifers" from killing guards or prisoners. Blackmun said such procedures were "not necessary as a deterrent." Inmates serving life sentences can still be given the death penalty if convicted of murder, he said.

Although 10 other states at one time had mandatory death penalties, all have been repealed or struck down by lower courts. Nevada's statute also has been repealed, but Shuman was sentenced before then. Two Alabama death row inmates are the only others known to have been facing execution under the law struck down by the court yesterday.

The intoxicated-juror case began after jurors told defense lawyers that as many as seven of them had been regularly drinking alcohol -- up to three pitchers of beer among four of them -- and using marijuana and were asleep for much of the trial.

Lower courts, citing federal rules against any action that might hinder internal jury deliberations, refused to hold a hearing to question the jurors about their behavior. O'Connor upheld those rulings.