The Supreme Court, signaling that it will closely monitor the application of death penalty laws in the lower courts, yesterday voided the death sentence imposed on a Georgia man who killed his wife and mother-in-law.

The 6-to-3 decision brought charges from dissenters that the court was embarking on a new phase of "second-guessing" lower court judges, and juries by assuming the task of "determining on a case-by-case basis" whether someone deserves the death sentence.

The court ruled that Georgia's death penalty statute, which was left intact, was being interpreted too vaguely by state courts.

A Georgia jury sentenced Robert Franklin Godfrey after convicting him of the two September 1977 shotgun murders.

Godfrey's wife, after a "heated argument" with her husband, had gone to live with her mother in a trailer. Believing the two to be conspiring against him, Godfrey showed up a few days later with his shotgun, aimed it directly through a window at his wife, and killed her. Then he entered the trailer and shot his mother-in-law.

Godfrey surrendered to the local sheriff.

Georgia's capital punishment law, which has been expressly approved by the Supreme Court, allows the death sentence for, among other things, crimes "outrageously or wantonly vile, horrible or inhuman in that (they involve) torture, depravity of mind or an aggravated battery. . . ."

The jury, without specifying its reasons, decided that the description applied to Godfrey's case. The Georgia Supreme Court agreed.

The U.S. Supreme Court, which in the past has largely confined itself to ruling on the validity of laws, yesterday reversed the Georgia decision as a misapplication of the law. Godfrey's crimes did not involve torture or severe beating, Justice Potter Stewart wrote, and were no more depraved than any other murder.

"There is no principled way to distinguish this case, in which the death penalty was imposed, from the many cases in which it was not," Stewart wrote. And that distinction is essential, he said, if death sentences are not to be imposed in an "arbritrary or capricious" manner struck down as unconstitutional by the Supreme Court in its previous decisions.

Justices Harry A. Blackmun, Lewis F. Powell and John Paul Stevens joined Stewart. Justices Thurgood Marshall and William Brennan concurred in separate opinions.

But Chief Justice Warren E. Burger and Justices Byron White and William Rehnquist dissented, saying the court's decision interferes with the prerogatives of the lower courts.

"Our role is not to peer majestically over the lower court's shoulder so that we might second-guess its interpretation" of valid laws, White and Rehnquist wrote.

"It is emphatically not our province to second-guess the jury's judgment or to tell the states which of their 'hideous' intentional murderers may be given the ultimate penalty," Burger wrote separately.

Brennan and Marshall, along with anti-death-penalty lawyers interviewed yesterday, said that the decision shows again that capital punishment cannot be implemented objectively. It is "doomed to failure" in that respect, the two justices wrote.

The court's division reflected its continuing inability to come together on many of the most emotional issues it considers, such as busing, and obscenity, in addition to the death penalty. Lawyers speculated that yesterday's decision would produce the same results in the lower courts as decisions in other controversial areas: confusion.

In other actions yesterday:

The court agreed to review a finding by the U.S. Court of Appeals in Washington that former president Nixon and his aides are liable for money damages in connection with the wiretapping of Morton Halperin, a onetime aide to Henry A. Kissinger. Halperin's phone was tapped in 1969 after Kissinger listed him as a suspect in leakage of national security information to the press. The Carter administration has asked the court to grant complete immunity from such damages to presidents and their immediate aides.

The court refused to consider an appeal by Iranian students from a lower court decision upholding Carter administration screening and deportation proceedings imposed last November after the taking of American hostages in Iran.

The court declined to consider a Minnesota man's unsuccessful effort to change his name to a number: 1069. Lower courts told Michael Dengler that his unusual request would contribute to the "dehumanization" of the individual. Dengler appealed, contending that the number 1069 represented his true identity.