More selectivity by Supreme Court justices, not Chief Justice Warren E. Burger's proposed national appeals court, would solve problems created by the court's heavy caseload, according to a review of the court's decision-making process.

The study, directed by New York University law school professors Samuel Estreicher and John Sexton, estimates that one-fourth of the 165 cases decided in the 1982-83 term did not involve nationally significant issues requiring high-court intervention. The research also included the nearly 2,000 cases that the court declined to review.

The study said that the court cannot continue accepting cases, especially in the criminal area, "merely to correct perceived errors" by lower courts and that it should let cases "percolate" longer in federal appeals courts before intervening.

Estreicher, who clerked for Justice Lewis F. Powell Jr., and Sexton, who clerked for Burger, said their study "undermines" the case for a national appeals court, which Burger has advocated for more than a decade, most recently two weeks ago in a speech to the American Bar Association convention.

Through a press officer, Burger yesterday declined to comment on the study.

His idea has been endorsed by a majority of the current justices and has received bipartisan congressional support, but legislation to create a new court never has advanced beyond subcommittee approval.

Rep. Robert W. Kastenmeier (D-Wis.), who introduced such legislation last year, said those considering the caseload problem have been "perplexed about how it should be resolved. This new study is on target in terms of assessing the question. It must be admitted that decisions in this area must be subjective."

Proponents have said that, because the issue attracts little attention outside the world of legal scholars and judges, Congress does not give it priority.

Another stumbling block is that some liberals have expressed concern that a new court would give the conservative-dominated high court the ability to accept and decide even more cases. The Reagan administration has declined to endorse the proposal until other steps are taken to reduce the justices' caseload.

In arguing for the new court, Burger cited an "avalanche of cases", saying the number coming to the high court has soared from 1,463 when Earl Warren presided as chief justice in 1953, to 5,100 last year, including about 3,000 prisoner appeals.

The number of opinions has more than doubled in 30 years to more than 150 in recent terms, about 50 more than most experts believe that the court can handle capably.

Burger's proposed court would be selected by the Supreme Court and comprise 13 judges, one from each of the federal courts of appeals. Whenever two or more appeals courts disagree on an issue, the high court could refer cases to the new court.

Burger said the new court could relieve the high court of more than one-fourth of its cases.

The 1,200-page NYU study, to be published this spring, as well as reports published last week by Hastings College of Law in San Francisco, conclude that proponents of a new court vastly overstate the high court's burden, that such a court will not remedy the workload and that the justices can solve the problem without creating a new court.

The NYU review, according to an executive summary released yesterday, said a new court was likely to create more work for the justices, because they would have to review cases before giving them to the new court and would retain review of the new court's decisions.

Burger said the idea of creating a new court has been studied and debated for more than a dozen years. "The time has come to act," he told the ABA.

But the NYU and Hastings reports call for more study.

"Where is the evidence," law professor Arthur D. Hellman asks, of conflicting appeals court decisions that must be resolved. Writing in the Hastings report, Hellman said that, while more cases are coming to the Supreme Court, the vast majority are frivolous and take virtually no justice's time before being dismissed.

Without further study, he said, there is a "risk of pursuing mischievous 'solutions' to problems that exist only in the mind of the beholder."