The Supreme Court ruled yesterday that the Constitution forbids states from executing the insane, saying that such executions would be cruel and unusual punishment and would amount to the "barbarity of exacting mindless vengeance."

The 5-to-4 ruling, written by Justice Thurgood Marshall, cast doubt on the constitutionality of procedures in 38 states, including Maryland and Virginia, to guard against executions of death row inmates who claim that they are insane and do not understand what is happening to them.

All states prohibit execution of the insane, in part on the premise that it is futile to punish someone who does not understand that he is being punished. But, until yesterday, the high court had not required such a prohibition and had not ordered adequate procedures be adopted to protect the rights of death row inmates claiming they are insane.

Marshall said the Eighth Amendment's ban on "cruel and unusual punishment" prohibits executions of insane prisoners who did not know what was happening to them or why, regardless of whether those prisoners were sane at the time of their crimes or during their trials.

"Whether its aim be to protect the condemned from fear and pain without comfort of understanding, or to protect the dignity of society itself from the barbarity of exacting mindless vengeance," Marshall said, the prohibition "finds enforcement in the Eighth Amendment."

The court, acting in the case of a condemned killer in Florida, said the state's procedures were unconstitutional because they did not give the prisoner a fair chance to argue that he was insane, allowing the decision on insanity to be made exclusively by the governor.

The court ordered a new hearing for Alvin Ford, who was convicted of murdering a Ft. Lauderdale policeman during the robbery of a Red Lobster restaurant.

Ford's lawyers claimed that his 12 years on death row, during which he twice came within hours of execution, had driven Ford insane. They obtained opinions from two psychiatrists, including one who said that Ford was a paranoid schizophrenic and incompetent for exection.

The lawyers petitioned Gov. Bob Graham (D), who, following procedures outlined in the state's law, appointed three psychiatrists to examine Ford. The three examined him together for 30 minutes and, for different reasons, found him competent for execution.

Ford's lawyers were not allowed to participate fully in the process, and it was not clear whether Graham considered the opinions of Ford's psychiatrists.

"That this most cursory form of procedural review fails to achieve even the minimal degree of reliability required for the protection of any constitutional interest," Marshall said, "is self-evident."

"The commander of the state's corps of prosecutors," Marshall said referring to the governor of Florida, "cannot be said to have the neutrality that is necessary" to ensure an impartial fact-finding procedure.

"It is no less abhorrent today than it has been for centuries to exact in penance the life of one whose mental illness prevents him from comprehending the reasons for the penalty or its implications," Marshall concluded. He was joined in full by Justices William J. Brennan Jr., Harry A. Blackmun and John Paul Stevens, and in part by Justice Lewis F. Powell Jr., who cast the critical vote in the case.

Powell, in a concurring opinion, said no state "disputes the need to require that those who are executed know the fact of their impending execution and the reason for it."

That minimum standard of knowledge, Powell said, is required under the Eighth Amendment. "If the defendant perceives the connection between his crime and his punishment, the retributive goal of the criminal law is satisfied. And only if the defendant is aware that his death is approaching can he prepare himself for his passing. Accordingly, I would hold that the Eighth Amendment forbids the execution only of those who are unaware of the punishment they are about to suffer and why they are to suffer it."

Powell said he would not "require the kind of full-scale 'sanity trial' that Marshall appears to find necessary," but said the states should set up an "impartial officer or board" to review the cases and receive evidence to make sure "basic fairness is observed."

Justice Sandra Day O'Connor, joined by Justice Byron R. White, said the Constitution does not bar states from executing the insane. But O'Connor said that, since Florida law had prohibited such executions, constitutional due process required that Ford be given "minimal procedural protections" and a new hearing.

Justice William H. Rehnquist, joined by Chief Justice Warren E. Burger, dissented, saying that all states prohibited executing the insane and he found it "unnecessary to 'constitutionalize' the already uniform view that the insane should not be executed."

Psychiatrists yesterday said the ruling -- particularly Powell's statement that inmates, once cured of their mental problems, could be executed -- created profound ethical dilemmas for the profession.

"The really difficult ethical dilemma comes in treating someone found incompetent," said Dr. Howard Zonana, associate professor of psychiatry at Yale University. "You want to ease the pain of psychosis, but you also know that that is going to lead to his execution." He said psychiatrists "have discussed this a good bit, but there is no an easy answer."

In other rulings yesterday, the court:Ruled 6 to 3 that prosecutors may use incriminating statements defendants make to jailhouse informers so long as those informers do not elicit the statements. The case is Kuhlman v. Wilson. Ruled unanimously in Kimmelman v. Morrison that inmates have a constitutional right to ask federal courts to consider claims that they were denied the effective assistance of a lawyer in their state trials. Divided 5 to 4 in Smith v. Murray, ruling that a convicted Virginia killer's death sentence could not be judicially reviewed because his lawyer did not press a constitutional claim at the appropriate time.