An emotional Harrison A. Williams Jr. ended his defense before his Senate colleagues yesterday, declaring that he had been the victim of "grand masters of deceit" who had been given power to "swindle or frame whomever they chose."

As Williams finished, leaders of the Senate Ethics Committee began a methodical, point-by-point rebuttal that cast into doubt much of what the New Jersey Democrat had said during his more than 5 1/2 hours on the Senate floor.

The debate ended amid growing signs of restlessness in the Senate, which has devoted three full days to consideration of a resolution to expel Williams, who was convicted of conspiracy and bribery last year in the government's Abscam probe.

A vote on a move to censure the 62-year-old Williams, rather than expel him, is scheduled for Wednesday. An expulsion vote could come later that day or Thursday.

Yesterday's rebuttal of Williams' defense began almost apologetically. "The Ethics Committee has no desire to pursue or create a vendetta against Sen. Williams. It's preposterous," said Sen. Malcolm Wallop (R-Wyo.), chairman of the panel, which voted unanimously last summer to expel Williams. "We would have rather had the whole matter go away. The truth of it is, we simply could not."

Wallop and Committee Vice Chairman Howell Heflin (D-Ala.) then systematically attacked efforts by Williams, a 23-year Senate veteran, to portray himself as the victim of wrongdoing, rather than the perpetrator. They did so by using Williams' own words from tape recordings the government made in the case, and the opinions of federal judges who spent weeks studying the case.

The real issue, Wallop said, "is not, as Sen. Williams would have us believe, the conduct of others, but rather, the conduct of Sen. Williams himself."

To suggestions that Williams, the third incumbent senator in history convicted of a felony, be censured rather than expelled, Wallop said, "There can be no compromise with wrongdoing, bribery, influence peddling, conflict of interest and ethically repugnant conduct."

Williams misled the Senate on a series of critical points, Wallop said. There was "no evidence" that the FBI sought out Williams, as he had contended, Wallop said. "Quite the contrary, the evidence is clear that Sen. Williams became the subject of investigation through a self-selecting process."

As Wallop spoke, Williams slumped in his chair in the second row, his forehead resting in his left hand.

The New Jersey senator began the day by denying he was considering resignation. He told a noon press conference: "I just can't conceive of it coming to that . . . if I'm successful in making the senators know what was on my mind and what I did and what I didn't do."

But his colleagues appeared to be losing patience with the case. It took 65 minutes yesterday before Majority Leader Howard H. Baker Jr. (R-Tenn.) thought there were enough senators present to begin.

Throughout the proceedings, there was much yawning, rubbing of eyes and shifting around in seats. On the desk of each senator was a one-page copy of Senate Resolution 204, which recommends that Williams be expelled for "ethically repugnant" behavior, and a loose-bound booklet summarizing the evidence in the case.

Few senators were seen opening the booklet. A few appeared to have difficulty even staying awake. Others slipped in and out of the chamber throughout the afternoon.

Williams was far more forceful and emotional during yesterday's presentation than he had been on Thursday, when he opened his defense with a rambling speech.

His face turned red with rage as he began. His usual monotone baritone voice rose and fell dramatically, at times quivering. "The most tragic irony of this sordid fantasy of Abscam is that the FBI, the judicial system, the Congress and I became victims of grand masters of deceit who were given almost indiscriminate and uncontrolled use of the power, prestige and influence of the FBI in order to swindle or frame whomever they chose," he said.

Still, he offered "almost an apology" for his actions, saying he could "kick myself" for meeting not once or twice with the undercover con men, but seven times.

Williams said the long Abscam investigation had "warped and twisted" him into a "synthetic creation" and made him falsely appear full of greed.

"The situation I face is unique in the history of our nation," he continued. "Never before have the employes of the executive branch tried to frame a member of Congress. Never before in the history of our great nation has a senator been convicted of a created crime."

When he ended, 86 minutes later, he was apologetic and looked emotionally spent. "I am sorry, sorry indeed, it came to this," he concluded.

Williams was convicted of agreeing to use his office in exchange for a hidden 18 percent share in a mining venture that was to have been financed by a $100 million loan from an undercover agent posing as an Arab sheik. He received a three-year prison sentence and a $50,000 fine.

Wallop said much of what Williams and others have said in his defense so far in the debate was false, misleading or irrelevent. He said statements by Williams "that he knew the stock to be worthless" were inconsistent with his testimony at his trial and before the Ethics Committee.

During last week's debate, Sen. Daniel K. Inouye (D-Hawaii) said that no senator had ever been expelled for any reason other than treason or disloyalty. Wallop said Senate committees had recommended that senators be expelled for bribery-related matters on five other occasions; two resigned, two were cleared of the charges and one was defeated for reelection before the matter came to a vote.

Wallop also discounted a report by Roger Shuy, a Georgetown University linguist who maintained that Williams' words on the government's tapes did not convey his true intentions. Williams, Wallop noted, did not use Shuy's report in his criminal trial and had never questioned the accuracy of the tapes--"yet we are now being told that we cannot understand the words and deeds of one of our colleagues, as recorded on these tapes, without the aid of the linguist to interpret them for us."