Anti-abortion leaders testified yesterday that Supreme Court nominee Sandra D. O'Connor has failed in three days of Senate confirmation hearings to pass their test of acceptability, then mounted a graphic attack on abortion, complete with descriptions of maimed fetuses and references to Nazi medical experimentation.

Both liberals and conservatives rallied to O'Connor's defense, however, indicating that her confirmation is in no danger and that the anti-abortionists will suffer one of their most important defeats when the Senate votes on confirmation, possibly next week.

Sen. Howard Metzenbaum (D-Ohio) attacked them for urging a nominee's rejection on a single issue. Minority Leader Robert C. Byrd (D-W.Va.) expressed his "enthusiastic" support and promised to "expedite" confirmation.

Chairman Strom Thurmond (R-S.C.) pronounced the committee "deeply impressed" by O'Connor and cautioned the anti-abortionsts that the hearing was not a "hearing on the abortion issue but on the fitness of a lady" for the Supreme Court.

Sen. Robert J. Dole (R-Kan.) said he did not think "we can subscribe to the position that she must repudiate" the Supreme Court's ruling legalizing abortion, which is what the anti-abortion groups wanted. "There is very strong support for her in this committee . . . . I was hoping that you would have found it possible to have a change of heart," he told the anti-abortion witnesses.

Dr. John C. Willke, president of the National Right to Life Committee, and Dr. Carolyn F. Gerster, an Arizonan and neighbor of O'Connor's who is vice president of the committee, were the principal anti-abortion witnesses yesterday.

"I came here prepared to tear up my testimony," Gerster said. " . . . I wish with all my heart I could support my fellow Arizonan." But because of her "pro-abortion" voting record as an Arizona legislator and her lack of criticism for the Supreme Court's 1973 legalization of abortion, Gerster and Willke said they opposed the nomination.

Willke said the abortion issue was like the slavery issue or the issue of racial justice: sufficient by itself to determine a nominee's acceptability.

"We believe that recognition of the right to life of unborn children is, likewise, a fundamental issue," he said. "Those who do not recognize this fundamental right should be considered disqualified for the federal bench."

"This is a once-in-a-century issue," said Willke, "such an abominable evil that, while being a single issue, it should be a disqualifying issue.. . . There is no middle ground. There is no compromise. A baby is either alive or dead."

Gerster said the Justice Department had misled President Reagan about O'Connor's abortion-voting record. "All of her votes" on the subject, except one, "have been consistently supportive of legalized abortion," Gerster told the committee. O'Connor was "an absolute adversary" while serving as Arizona Senate majority leader, she said.

Gerster, Willke and their principal committee supporter, Sen. Jeremiah Denton (R-Ala.), then began an hour-long attack on abortion that included descriptions of "decapitated fetuses" and an argument in favor of a constitutional amendment.

Other witnesses yesterday represented a cross-section of causes and interests.

Radio preacher Carl McIntire, his voice quavering with emotion, expressed fears that O'Connor's philosophy on federal judicial restraint would leave small religious colleges and radio stations at the mercy of unsympathetic state officials. McIntire-related institututions have been in court battles over their autonomy frequently in recent years.

The National Bar Association, a predominantly black lawyers' organization, the American Bar Association and the National Organization for Women expressed support for the O'Connor nomination. The American Judges Association described her as an "eminent jurist, with an impeccable professional and social background."

Arizona Gov. Bruce Babbitt, who appointed O'Connor to the Arizona appeals court, said the nomination was the most significant for the states in a generation because of her background as a state legislator and judge and her support for state judicial authority.