Two former Abscam prosecutors charged yesterday that convicted con man Mel Weinberg, who worked as the FBI's middleman in the undercover sting operation, actually operated a "scam within a scam," taking advantage of both the government and the targets of the investigation.

Edward J. Plaza and Robert A. Weir Jr., two assistant U.S. attorneys who worked in the Newark, N.J., office, told a House Judiciary subcommittee yesterday that they demanded to be and were removed from the case in June, 1980, because they objected to the way the investigation was being run. Weir is still an assistant U.S. attorney in Newark; Plaza has gone into private law practice.

Abscam was the name given the FBI scheme to use a phony Arab sheik to try to bribe members of Congress in return for favorable legislation.

One U.S. senator, six House members, and a number of local officials have been convicted in the case. The conviction of former representative Richard Kelly (R-Fla.) was vacated last month by U.S. District Court Judge William B. Bryant here on the grounds that Kelly was denied due process. The other convictions have generally been upheld, and the government is appealing the Kelly ruling.

Plaza said the public saw only the videotapes of members of Congress accepting bribes. What it did not see or hear, he said, was transcripts or recordings of meetings that preceded the offering of the bribes. Such evidence is usually presented in similar cases, but transcripts and recordings were not always made or transcribed during Abscam, he said.

Plaza and Weir charged that the usual investigative procedures were violated in Abscam, that the required reports on conversations between parties in the case were often ignored. Even when conversations were recorded, he said many were lost and others were not transcribed until after indictments were under way.

"The rules were totally different," Plaza said of the Abscam investigation. "This was an aberration, totally different from any other experience I had as a prosecutor."

He said telephone records showed that Weinberg had more than 80 unrecorded conversations with Mayor Angelo Errichetti of Camden, N.J., who was convicted in Abscam. "It became clear to us Mel Weinberg intentionally didn't record those conversations so he could 'scam' the agents and the government."

Plaza said there were "many unrecorded conversations and meetings," and as a result, there was "a lack of any kind of documentation" of those talks.

"That was one of the problems about Abscam--there were very few transcripts being made," Plaza said.

As a result, Weir said, the public and the courts have been denied all the facts about Abscam and what led up to the bribes.

Plaza said that after spending 15 months working on Abscam, he concluded that many of the cases were orchestrated by Weinberg who coached and put words in the mouths of the targets and apparently had his own priorities separate from the FBI.

"Abscam represents the selective use of technology to create an illusion of criminality. It was tantamount to prosecuting actors in a play for following a script . . . . Abscam is a perversion of the truth," Plaza said.

He said that at the same time Weinberg had negotiated a deal with the FBI to be paid large amounts of money for bringing in prominent politicians, he was also splitting bribes with some of the targets of the investigation and intercepting gifts intended for the phony sheik. Those gifts included three gold watches, a video cassette recorder, a microwave oven and color television sets.