The American Bar Association, rebuffing an unprecedented personal plea from U.S. Supreme Court Justice William H. Rehnquist, voted today to oppose creation of a new federal appeals court.

At the group's midwinter meeting here, the ABA's House of Delegates also passed a resolution aimed at toning down the lawyers' conflict with doctors over the handling of medical malpractice cases. But the group unanimously adopted a report by a special ABA committee that rejected all the American Medical Association's major proposals for restricting malpractice suits.

The proposed Intercircuit Panel, which the ABA voted against today, would add a layer of judicial review between the 13 circuit courts of appeal and the Supreme Court.

It has been a pet project of Chief Justice Warren E. Burger, who sees the tribunal as a means of reducing the high court's workload and urged the ABA to endorse it at the group's meeting last year.

Rehnquist, noting that the Supreme Court currently decides about 150 cases annually out of about 4,500 it is asked to review, said it is "simply indisputable" that "we need more national decision-making capacity."

An ABA spokesman said Rehnquist's appearance marked the first time a sitting Supreme Court justice had participated in a House of Delegates debate on a pending proposal.

Nonetheless, the 441-member house, siding with a majority of federal appellate judges, voted to oppose the creation of the "mini-Supreme Court." Supporters of the new court had hoped ABA backing would give new impetus to the proposal, which has moved slowly in Congress, in part because it has been opposed by the Reagan administration.

Under the latest measure, the court would be established for a five-year experimental period and consist of nine circuit court judges, who would be appointed by the Supreme Court to review circuit court and state supreme court cases. Most of the cases would involve conflicts between the circuits or interpretations of federal law.

Those who spoke against the court, including three federal judges -- D.C. Circuit Court Judge Patricia M. Wald, 2nd Circuit Judge James L. Oakes and 8th Circuit Senior Judge Floyd R. Gibson -- argued that, although some changes in the system are needed, the proposed court would actually increase the high court's workload and make litigation slower and more costly.

The proposed court, Gibson said, would "only result in confusion and disruption of the administration of justice."

On the medical malpractice issue, the ABA adopted an amendment to its special committee's report stating that both the legal and medical professions "should avoid any efforts to polarize the discussion."

The new language, said Richmond lawyer George C. Freeman Jr., will "go a long way to allay the increasing concerns of many of us" about a perception that the ABA is "opposed to fundamental reforms and . . . engaged in an all-out war with the American Medical Association."

If the public had to choose between doctors and lawyers, he said, "we would come out on the wrong side of the choice."

The 44-page committee report, adopted as ABA policy today, rejected doctors' bid for "special treatment," saying that "the medical profession's perception that the medical malpractice insurance issue has reached a 'crisis' stage that merits major changes in tort law and procedure appears to be unjustified."

The AMA is pressing, among other things, for limitations on punitive damages, on recovery for pain and suffering, and on contingent legal fees, under which lawyers collect a share of any money won.

ABA President William W. Falsgraf, saying that "the lawyers . . . are speaking for the injured parties of this country," said that under the AMA proposals, "the medical profession, unlike any other segment of society, would be exempted from important provisions" of the law.