The American Bar Association's Committee on the Judiciary yesterday took the apparently unprecedented step of declaring that a Reagan administration nominee to a federal appeals court is unqualified for the appointment because of a lack of "personal integrity."

The nominee, Commerce Department General Counsel Sherman E. Unger, is being considered by the Senate for confirmation to the new U.S. Court of Appeals for the Federal Circuit.

In a report submitted to the Senate Judiciary Committee, the ABA panel accused Unger of a series of ethical violations while in private practice, including "filing false affidavits," engaging in improper contacts with judges and parties in cases, and improperly receiving legal fees in a bankruptcy case.

The report also included a statement that Unger was relieved of his job as general counsel in the Department of Housing and Urban Development in the Nixon administration because he was "untrustworthy and allowed his self-interest to interfere with his judgment."

The 14-member ABA panel was unanimous in its recommendation, according to William T. Coleman Jr., who spoke for the ABA yesterday.

Unger defended himself in a statement presented to the Judiciary Committee yesterday expressly denying some of the allegations and explaining others as mistakes made when he was inexperienced in bankruptcy practice.

Experienced observers of the judicial nominating process said they could not recall any similar recommendation by the ABA involving an appeals court appointment, though four federal trial court judgeship candidates in the past few years have received "unqualified" ratings because of questions about their integrity.

The Reagan administration, which was informed months ago of the allegations, is standing behind Unger's nomination.

Unger is supported by a number of prominent Republicans, including Commerce Secretary Malcolm Baldrige, Sen. John W. Warner (R-Va.), Sen. Bob Packwood (R-Ore.) and former senator Hugh Scott (R-Pa.), all of whom appeared before the committee to vouch for the nominee's integrity and qualifications.

Two Democrats, former White House counsel Lloyd N. Cutler and former attorney general Griffin B. Bell, also appeared on his behalf yesterday.

Unger is the first nominee to the new appeals court, created by combining the old Court of Claims andCourt of Customs and Patent Appeals. It has the status--just below the Supreme Court--of the other 12 courts of appeals around the country, but its jurisdiction is by subject rather than geography.

The ABA panel said Unger had received generally high marks for his technical and intellectual prowess as an attorney.

But it said he was "not qualified because he lacked the personal integrity and judicial temperament required of a federal judge," Coleman testified.

Coleman also said Unger, in a number of interviews with panel members, had failed to reveal facts concerning the alleged ethical violations and had changed his stories when confronted with documentary evidence. Coleman said Unger's "failure to be forthright with the ABA committee not only hampered our ability to conduct the investigation but also reflected adversely on his candor and personal integrity."

Coleman said the most serious allegation was that Unger "frequently engaged in improper" private meetings with judges in cases without notifying or inviting opposing lawyers.

Such ex parte contacts are strictly prohibited by both judicial and lawyer codes of ethics.

The report included documents on one such alleged incident in 1975 when, in the course of representing a client in a bankruptcy proceeding, Unger met with the judge privately to seek the reversal of an unfavorable order.

Unger, in his statement, said the "ex parte contact was at the request of the judge and in conformance with the then-existing practice in bankruptcy proceedings in the Southern District of Ohio," where Unger practiced at the time.

Coleman said the evidence, including sworn testimony by the now-deceased judge in the matter, contradicted Unger's claim that the judge initiated the contact.

The ABA report also said that Unger filed "untrue affidavits in certain bankruptcy proceedings" stating he had no "relation" to any of the parties when, in fact, he held joint business interests with some of them.

"I admit the affidavits could have been more precisely written," Unger said yesterday, noting that they were prepared during his first bankruptcy case.

Unger denied that he was fired from his HUD job. But a statement by Richard C. Van Dusen, then undersecretary of the department, said Unger was fired when he refused to resign in 1970.

The discharge followed efforts by Unger to get an appointment as chairman of the Federal National Mortgage Association, the Van Dusen statement said. When Unger learned that someone else was slated for that post, it added, he wrote a legal opinion declaring his competitor ineligible.

"This was a source of considerable embarrassment," Van Dusen said. "Mr. Unger's actions, in which, in my opinion, his own self-interest led him to act in a manner contrary to the expressed positions of the Secretary, caused the Secretary and myself to conclude that he could no longer serve effectively as General Counsel of HUD."

Another allegation in the report concerned Unger's "termination" from a job as vice president of American Financial Corp.

The ABA produced affidavits alleging that Unger "threatened to commit suicide and blame his death on the officers" of the corporation.

"These allegations are, of course, false and I have always denied them," Unger's statement said.

Coleman, a prominent Washington attorney, was secretary of transportation in the Ford administration.