The special counsel to the House Ethics Committee's Abscam investigation resigned in protest late Tuesday after the committee rejected, on a rear party-line vote, his recommendation to file misconduct charge against Rep. John P. Murtha (D-Pa.), according to sources.

E. Barrett Prettyman Jr., the committee's special counsel, said yesterday that he had submitted his resignation after the committee vote on the Murtha matter Tuesday, but he declined to discuss his reasons, saying only that they were outlined in a letter to committee members.

The committee office put out a statement saying the members dismissed the Murtha matter after finding, "on the basis of the evidence presented, no reason to issue a statement of alleged violation." Chairman Louis Stokes (D-Ohio) said yesterday that he was bound by committee rules not to discuss executive-session votes. He also declined to say why Prettyman resigned. "He would have to tell you his reasons," Stokes said.

Murtha was never charged in the Abscam investigtion, although he was named as a coconspirator in the indictment against then-Reps. Frank Thompson Jr. (D-N.J.) and John M. Murphy (D-N.Y.) and testified against them at their bribery trial last fall.

On the FBI's undercover videotapes, Murtha is seen talking to agents about a $50,000 payoff for providing immigration help for a phony Arab sheik. He refused the money, although he did say he might be interested after he got to know the would-be givers better. Sources said Prettyman had prepared several charges of ethical misconduct against Murtha, but all were rejected.

One committee source said yesterday that the key vote was "a party-line vote." The committee is made up of six members of each party, so seven votes are needed to file a charge. A Murtha aide said yesterday that he knew nothing about Prettyman's resignation.

During the Thompson-Murphy trial, Murtha testified that Thompson wanted him to get two more congressmen involved in a deal that could head to investment in his district. "But all we would have to do is help these two Arabs get into the country sometime in the future," Murtha said. He said he would be interested because his district had high unemployment.

Murtha also testified that Thompson told him there would be "some walking-around money available for the three congressmen involved." He understood that to mean $50,000 in cash, he said. Murtha also said he knew that money might be offered to him before he went to a meeting with the supposed representatives of the sheik.

Prettyman was hired by the committee early last year to run an investigation parallel to the justice Department's. After House members were convicted in the Abscam cases, Prettyman prepared disciplinary charges that led to the expulsion of Rep. Michael (Ozzie) Myers (D-Pa.) and the resignation of Reps. John Jenrette (D-S.C.) and Raymond F. Lederer (D-Pa.). He was said to be checking mentions of other congressmen on the Abscam tapes at the time of his resignation.