The Supreme Court refused yesterday to widen the Constitution's protection against compelled self-incrimination, ruling 5 to 4, that a juvenile in police custody should have asked for a lawyer rather than his probation officer.

The court acted in a California murder case in which "Michael C.," 16, was informed of his so-called Miranda rights to remain silent or to consult an attorney.

A Van Nuys police interrogator told the youth, "I can't get a hold of your probation officer right now. You have the right to an attorney."

Michael, who had been told by his probation officer to contact him if he ever had "a police contact," was suspicious. "How I know you guys won't pull no police officer in and tell me he's an attorney?" he asked.

Within a few moments, however, Michael, informed once again of his Miranda rights, said, "Yeah I want to talk to you," he said.

He then made statement and drew sketches that incriminated him in the 1976 murder of Robert Yeager, 64, a Hollywood film publicist who was shot in the head when he opened the front door of his home to put out the trash.

In juvenile court, counsel for Michael sought to suppress the statements and sketches on the ground that they had been obtained in violation of the Supreme Court's 1966 ruling that interrogation must cease when Miranda rights are invoked.

By asking to see his probation officer, the counsel argued, Michael had invoked the Fifth Amendment privilege to remain silent and had done so as if he had requested the aid of an attorney.

The judge denied the suppression request but was overruled by a divided California Supreme Court.

Michael had asked to see his probation, the majority said. This negated any possible willingness to discuss his case with the police, it held.

In making the ruling, the state tribunal emphasized that for Michael, the probation officer was "a trusted guardian figure who exercises the authority of the state . . . and whose duty it is to implement the protective and rehabilitative powers of the juvenile court." Michael had a long record and had been on probation since he was 12.

The majority also said that, as a law enforcement agent, the probation officer would act to protect Michael's rights just as a lawyer would have.

"It is fatuous to assume that a minor in custody will be in a position to call an attorney" or trust the police to call one for him, the court said.

These views were rejected yesterday when the Supreme Court reversed the state tribunal with an opinion by Justice Harry A. Blackmun.

"Whether it is a minor an adult who stands accused, the lawyer is the one person to whom society as a whole looks as the protector of that person in his dealings with the police and the courts," Blackmun wrote.

For this reason, he said, the court fashioned in the Miranda case "the rigid rule" that an accused's request for an attorney is, of itself, an invocation of Fifth Amendment rights.

By contrast, Blackmun said, a probation officer is not trained in the law, is not a trained advocate and "is the employe of the state which seeks to prosecute the alleged offender."

He said he also could not attach to Michael's request to see the probation officer the "overwhelming significance" attached to it by the state tribunal: that it was a request to remain silent.

In a dissenting opinion, Justice Thurgood Marshall wrote that the Miranda rules require a halt to interrogation "whenever a juvenile requests an adult who is obligated to represent his interests." On its face, he said, Michael's request for his probation officer was an assertion of his Fifth Amendment rights. Justices William J. Brennan Jr. and John Paul Stevens signed the opinion.

Justice Lewis F. Powell Jr., dissenting separately, said he was not "satisfied that this particular 16-year-old boy, in this particular situation, was subjected to a fair interrogation free from inherently coercive circumstances."

The Court handed down other decisions.

LUGGAGE SEARCHES

The Court ruled, 7 to 2, that unless there is an emergency, police are required under the Fourth Amendment to get a warrant before searching luggage taken from an automobile that they had properly stopped to search for contrabrand.

The ruling upheld an Arkansas Supreme Court decision that the suspect occupant of the car and the luggage should have been taken to police headquarters so that a magistrate could decide whether to issue a warrant.

Justice Lewis F. Powell Jr. wrote the opinion for the court. Justice Harry A. Blackmun, dissenting, foresaw problems, such as a suspect container in a car trunk being not a suitcase, but "an orange crate, a lunch bucket, an attache case, a duffel bag, a cardboard box . . ." His opinion was signed by Justice William H. Rehnquist.

SOCIAL SECURITY

Each year, erroneous overpayments are made to Social Security recipients under the old age, survivors' or disability insurance programs. The law authorizes the government to recoup the overpayments by reducing future payments in the necessary amounts.

The practice of the Social Security Administration has been to allow an allegedly overpaid recipient to file a written request seeking reconsideration of the purported overpayment or to ask in writing for a waiver of recovery.

In a unanimous decision, the court held that only recipients who file a written waiver request - not those who merely request reconsideration - are entitled t a pre-recoupment oral hearing.

HOSPITAL

In another 9-to-0 ruling, the court held that a union can solicit hospital employes in lobbies, cafeterias and gift shops - but not in the corridors and sitting rooms that adjoin or are accessible to patients' rooms and operating and treatment rooms.