Supreme Court Justice William H. Rehnquist, in an unusually harsh attack on his colleagues, yesterday accused the court of making a "mockery" of criminal justice by undermining the implementation of the death penalty.

The court has done this by encouraging endless appeals and delays of capital punishment sentences, he said, and by surrounding capital defendants with so many procedural protections that it has become almost impossible to execute anyone who wants to live.

The result has frustrated the public's desire for "retribution," he said, and contributed to the rise of vigilantism. The court has ruled the death penalty constitutional, he said, but prevents it from being carried out.

"I believe we have in our judicial decisions focused so much on controlling the government that we have lost sight of the equally important objective of enabling the government to control the governed," he said.

In the past few years, more than 700 people have been sentenced to death but only one of them -- John Spenkelink -- has been executed against his will.

Rehnquist's comments, similar to a get-tough-on-criminals speech recently delivered by Chief Justice Warren Burger, was part of a written exchange among four justices over a decision yesterday not to review a murder conviction from Georgia. The defendant, Wayne Carl Coleman, was convicted in 1973 of murdering six members of a family after raping and torturing some of them. He has been filing appeals and habeas corpus petitions ever since and Rehnquist speculated that he can go on doing this interminably because of Supreme Court and lower court procedures.

Rehnquist said he wanted this case and others like it reviewed by the court to resolve at one time all the issues a defendant can raise. This would end the appeal process and, if the defendant loses, speed his death.

Justices John Paul Stevens, Thurgood Marshall and William J. Brennan, voting not to review the Coleman case, took strong exception to Rehnquist's proposals. Surely, Marshall wrote, no member of the court would "want to see an innocent individual put to death" because of an unconstitutional conviction overlooked by cutting off the appeal process.

Rehnquist's view of what the court has done with capital punishment -- if not his remedy -- is shared by meny advocates of the death penalty. The court has held capital punishment constitutional, but it has said that the states must be much more careful in implementing it than in meting out other punishment. The justices in some cases have ruled state capital punishment laws too arbitrary or too rigid.

Rehnquist called this line of decisions "tinkering," which has enabled defendants to delay with appeals and petitions the day of execution. The rulings have "sent a signal to the lower state and federal court that the actual imposition of the death sentence is to be avoided at all costs."

". . . I do not think that this court can continue to evade some responsibility for this mockery of our criminal justice system," Rehnquist said. ". . . This court, by constantly tinkering with the principles laid down in the five death penalty cases decided in 1976, together with the natural reluctance of state and federal habeas judges to rule against an inmate on death row, has made it virtually impossible for states to enforce with reasonable promptness their constitutionally valid capital punishment statutes.

"When society promises to punish by death certain criminal conduct, and then the courts fail to do so, the courts not only lessen the deterrent effect . . ., they undermine the integrity of the entire criminal justice system."

". . . There can be little doubt," Rehnquist wrote, "that delay in the enforcement of capital punishment frustrates the purpose of retribution . . . San Francisco experienced vigilante justice during the Gold Rush in the middle part of the last century; the mining towns of Montana experienced it a short time later;

"And it is still with us as a result of the series of unsolved slayings of Negro children in Atlanta."

"In the nation's capital," Rehnquist wrote, "law enforcement authorities cannot protect the lives of employes of this very court who live four blocks from the building in which we sit and deliberate the constitutionality of capital punishment [a reference to the recent murder of a Supreme Court messenger.]"