Chief Justice Warren E. Burger said today that the nation's judicial system is no longer "capable of handling all the burdens" placed upon it and should begin shedding responsibility for conflicts that can be settled by arbitration or mediation.

He recommended that legal controversies such as divorce, child custody, adoption and landlord-tenant matters be removed from the courts to relieve a crushing caseload burden. At the same time, he said, lawyers should begin training to conciliate disputes instead of litigating them.

Burger's suggestions came in his annual State of the Judiciary speech to the American Bar Association meeting here. He blamed his audience, lawyers and judges, for resisting out-of-court techniques for resolving disputes because they feared for their power or livelihoods.

Another cause of the crowded, expensive and time-consuming legal system, he said, were increasing demands for "legal entitlements."

"Remedies for personal wrongs that once were considered the responsibility of institutions other than the courts are now boldly asserted as legal entitlements," he said.

"The courts have been expected to fill the void created by the decline of church, family and neighborhood unity."

Burger's suggestions were not new; the "litigation explosion" has been a central concern of the legal profession for a decade. But the point of his speech appeared to be that despite all the talk, little has been done, and he called on the ABA, with 280,000 members the nation's largest professional association, to push for arbitration and mediation.

The speech, which received polite applause from the 900 in the audience here, is unlikely to provoke much controversy, in contrast to Burger's hard-line law-and-order address to the same group last year.

Burger quoted statistics suggesting that by the beginning of the next century, the federal appellate courts will decide a million cases per year and will need 5,000 judges, a tenfold increase, and 1,000 volumes to handle their efforts each year. He cited the increasing number of cases that go on for months, years and even decades, leaving other litigants "standing in line" and "denied access to the courts."

Arbitration, mediation and conciliation are ways of clearing the dockets, he said. But he said that lawyers and their clients, trained in the adversary system, were too combative and too concerned about winning rather than settling disputes.

"We need to consider moving some cases from the adversary system, from the trial by battle in the courtroom, to administrative processes or to mediation, conciliation and especially arbitration.

"Divorce, child custody, adoptions, personal injury, landlord and tenant cases," he said, "are prime candidates" for such processes, on a voluntary basis.

He said any new approaches must be realistic in order to succeed. "If a system of voluntary arbitration is to be truly effective, it should be final and binding."

He stressed, however, that no one should be forced into binding arbitration. But once they are in it, he said sanctions will be needed to "discourage further conflict." A party that goes to court unsatisfied with an arbitration should be made to pay the costs and attorneys fees unless he increases his monetary award by at least 15 percent.