The murder trial of Dr. William B. Waddill, which ended in a mistrial here late Friday, was not only emotion-laden, but also raised a host of unusual legal questions.

Waddill, a 42-year-old obstetrician with one of the largest practices in southern California, was charged with killing an infant girl born to an 18-year-old high school student after a saline (salt) solution abortion at a local hospital March 2, 1977.

After a 16-week trial, the nine-man, three-woman jury deliberated for 11 days and then announced it was hopelessly deadlocked. The judge declared a mistrial.

The reason the jurors gave for not being abel to reach a verdict was that the judge, James K. Turner, a veteran of 23 years on the bench and well-respected in legal circles, had confused them by giving them two definitions of death.

In his original instructions he told them death was "the permanent disappearance of all vital signs."

However, in an apparent unprecedented move, after they had deliberated for eight days he withdrew that definition and told them death was "a total and irreversible cessation of brain function."

The prosecution's key witness, Dr. Ronald Cornelsen, said he saw Waddill strangle the infant, which had displayed the "vital sign" of gasping. But he and several other doctors had testified that "brain death" could occur when only the brain stem, which controls the primitive functions of breathing and heartbeat, is functioning.

The jurors asked Turner to define these terms but the judge said that was their responsibility.

Waddill maintained from the beginning that the baby was "clinically dead," meaning only the brain stem was functioning, when he arrived at the hospital.

One of the jurors, Kathie Davis, said the jury had voted 8 to 4 for conviction before the judge gave the jurors the second definition, but the vote then changed to 9 to 3 for acquittal. The final vote on the seventh ballot was 7 to 5 for acquittal.

Veteran court observers said they could not recall a judge changing jury instructions after the jury had begun deliberations.

The reason Turner gave the new definition was that at the last moment defense attorneys Charles Weedman and Malbour Watson cited a Los Angeles murder case in which the appellate court had used the second definition, taken from the Health and Safety Code rather than the Penal Code, to define death.

Ironically, it was the prosecutor who reminded the defense attorneys of the Health and Safety Code definitioN.

Watson told reporters that prosecutor Robert Chatterton mentioned to him and Weedman another case he was involved in that turned on the safety code definition and the defense pounced on the information.

In another apparently unprecedented action, Turner ordered the jury sequestered after it had deliberated two days because of an "avalanche of publicity" and then two days later, after the jury complained about living conditions, ordered it unsejuestered and allowed the jurors to go home.

Although Deputy District Attorney Chatterton had said he would ask for a new trial in the event of a mistrial, he said Friday he would make that decision "within a week."

The case drew widespread attention from antiabortion organizaitons, such as the Right-to-Life group, which demonstrated at Waddill's preliminary hearing and packed the courtroom during the long trial.

The groups apparently thought that Waddill, in allegedly strangling a baby after an abortion he had performed resulted in a live birth, dramatized their belief that abortion in itself is murder.

It case also attracted the attention of the medical profession, since it is believed to be the first time a physician has been charged with first degree murder under the so-called "child abuse" section of the California Penal Code.

Although Waddill was originally accused of strangling the baby, Chatterton asked and Turner agreed over the objections of defense attorneys, to instruct the jury that it could find Waddill guilty if the jurors determined that he had a legal duty to administer "heroic" emergency treatment to the baby and that he had failed to do so.

The statute, section 273, formerly applied to persons with children in their custody who had not harmed the children themselves, but who had allowed harm to come to them by their inaction.