The Supreme Court, quarreling openly over a decision to hear a case, agreed yesterday to decide whether the voluntary confession of a mentally ill defendant can be used against him.

The court, in an unusual move, also asked on its own initiative for arguments on whether mentally ill people are capable of voluntarily waiving their rights under the 1966 Miranda ruling, which established the rights of suspects to remain silent and to have a lawyer present during questioning.

The addition of the Miranda inquiry raises the possibility that the court may be considering a further narrowing of the ruling, at least in terms of its application to confessions by the mentally ill or persons temporarily impaired by drugs or alcohol.

The case, Colorado v. Connelly, began in August 1983, when Francis Barry Connelly approached a uniformed patrolman on a downtown Denver street and said he had "killed someone." The officer promptly advised him of his rights, under the Supreme Court's Miranda decision, to remain silent and to have an attorney.

Connelly said he understood those rights and then said he had killed a 14-year-old girl with whom he had been traveling. He took police to the murder site and described the crime.

Denver police, who had discovered an unidentified girl's body a few months earlier at that spot but had no clues in the case, charged Connelly with murder.

Connelly later told a psychiatrist that the voice of God had ordered him to return to Denver from Boston and confess to the crime or commit suicide. The psychiatrist testified that Connelly was suffering from paranoid schizophrenia and hallucinations and was unable to make a free and intelligent decision about whether to confess.

The Colorado Supreme Court said Connelly's initial statement about killing the girl, though voluntary, could not be used against him without violating his rights to due process. The statements made after receiving his Miranda warnings could not be used without violating Miranda.

The Denver prosecutor appealed only the due process section of the ruling and said it was unnecessary to consider anything relating to the Miranda ruling.

The high court insisted on also hearing arguments about the state court's Miranda analysis.

This sparked an unusual if not unprecedented response from Justice William J. Brennan Jr., who said the court appeared to be acting as "not merely the champion, but actually an arm of the prosecution."

Brennan said the court's action "is yet another instance supporting the concern that the court shows an unseemly eagerness to act as the adjunct of the state and its prosecutors" in assuring convictions. "Today, the court takes the unprecedented step of rewriting a prosecutor's [appeal] for him, enabling him to seek reversal [of the Colorado court ruling] on a ground he did not present himself."

Brennan, joined by Justice John Paul Stevens, emphasized that the prosecutor specifically said the controversial Miranda decision, recently called "infamous" by Attorney General Edwin Meese III, was "not an issue" in the appeal.