He is a lonely, familiar figure, dressed in gray. Several times each day in the last several months he would enter the Senate, where he has worked for 23 years, quietly cast his vote, then leave, barely glancing sideways.

His colleagues would be gathered in small knots, chatting among themselves. Yet none approached Harrison A. Williams Jr. (D-N.J.), the only senator convicted of a criminal offense in 76 years, and he did not approach them.

An immensely personal institution, the Senate operates on unwritten rules. It holds a senator in trouble at arm's length. Williams understands the folkways of the place and says this does not bother him.

"I don't raise this business with the members, and they don't raise it with me," he says. "Otherwise, we get along fine."

More than two years have passed since Williams was indicted for bribery in the Abscam case, seven months since he was convicted, four months since the Senate Ethics Committee unanimously voted to expel him.

During that time, one-third of the membership of the Senate has changed. Yet Williams, once a power in the body, was at work every day until the session ended, attending committee meetings, voting, meeting with constituents and preparing his defense.

"You have a sense he has a lot on his mind," said Sen. Edward M. Kennedy (D-Mass.). "But at committee meetings he is attentive and well-briefed. He has handled this with dignity."

Months ago, Senate Minority Leader Robert C. Byrd (D-W.Va.) took Williams aside, told him "you've got a rough road ahead," and suggested that, for his good and that of the party, he resign. Williams, 61 and gray at the temples, says he never seriously considered resigning.

"I was set up. They had no reason to go after me in terms of my conduct," he says. "If you were set up and you knew you had done no wrong, would you resign? You know damn well you wouldn't. You'd fight. And that's what I'm doing. This is the most important fight for a principle that I've ever been in, and it has a tremendous meaning for the country and the left."

This gets to the heart of what keeps Williams going as he prepares to fight his expulsion from the Senate, according to those who know him best. He says he genuinely believes he is innocent. He refuses to admit he did anything wrong. He insists he is the victim, not the perpetrator of a crime.

Williams, a former chairman of the Labor and Human Resources Committee, has never been a dynamic figure. He is a slight man with a disheveled, low-key manner, a man easy to underestimate. His most distinguishing features are his large, bushy eyebrows.

For months, Williams' fight was a lonely one. The Ethics Committee voted, 6 to 0, in recommending that the Senate expel him. No senator rose to defend him publicly.

To outward appearances, Williams did not seem to notice. "He's held up amazingly well," one New Jersey friend said. "He gives the appearance of being a pushover, but he has a lot more fortitude than people give him credit for. Any public figure who can overcome alcoholism like he did in the early 1970s and survive in public life has to have an inner strength."

Asked recently how he has handled the ordeal, Williams broke into a broad grin and said:

"I drink a lot more than I did before."

"What are you drinking?"

"Coffee," he replied with a chuckle, enjoying the joke on his past problems. "I haven't been smoking more. I still have the same cigarette ration. I sleep well. I eat well."

Months of lonely combat have left a mark. A bunker mentality set in at his office. The senator and much of his staff have become suspicious of the media.

With few public allies, Williams, a liberal Democrat, began an unlikely flirtation with two groups that had come to his defense, the Church of Scientology, a controversial religious sect, and the National Democratic Policy Committee, a creation of Lyndon H. Larouche Jr., a right-wing ideologue.

"I don't think in past years he would have associated with groups like this," said one former associate, "but he was down to the wire, and these people seemed to be the only ones who were paying any attention to his case."

In early fall, Williams changed lawyers and press secretaries for the third time in recent months. The new defense attorneys took the offensive on what Williams calls "the third front" with a civil suit against the Justice Department, which Williams maintains entrapped him in the Abscam case.

Press secretary Walter Gold, a former Nixon advance man and former Washington Star reporter, prepared materials to acquaint other senators with his boss' side of the case. He also gave the senator a short course in media image-making.

"What we've done is to try to make him understand that the press, especially TV, only has so much time," Gold says. "You have to make your point fast so they can get it on a 90-second spot on the air, then go back and explain it. He has rehearsed to a point. He's gotten quite good, really."

The Senate resolution to expel Williams initially was scheduled to come to the floor Dec. 3. On Nov. 28, St. Paul's Seven Day Christian Church in Montclair, N.J., held a rally in Williams' honor.

"I felt it was critical to give a show of support from the black community. 'Pete' Williams has been an absolute advocate for everything the black community has tried to do during the last two decades," the Rev. Buster Soaries said. "The black community also knows something about the FBI. We're not willing to write someone off just because the FBI has tried to make them look like a criminal.

"We know what the FBI did to Martin Luther King. We know the FBI never found who shot Vernon Jordan. We know how inept the FBI was in finding the killer of the black children in Atlanta. And yet it spent tens of thousands of dollars trapping Pete Williams, whose only crime was to support Teddy Kennedy for president and express a desire to run for governor of New Jersey," Soaries said.

Soaries said the senator's wife, Jeannette, was "overwhelmed" by the rally. "Our prayer that day was that a vote not happen this week."

Meanwhile, support for Williams began surfacing in the Senate. The first voice came from an unlikely source, conservative Sen. Orrin G. Hatch (R-Utah).

Hatch told The Newark Star-Ledger that the FBI had violated Williams' constitutional rights. "The executive branch should never be able to act in this outrageous way to violate the due process rights of any member of a separate branch of government," Hatch said.

Then Sen. Daniel K. Inouye (D-Hawaii) volunteered to represent Williams on the Senate floor. The expulsion debate was delayed until early next year to give Inouye time to prepare the case.

Privately, some members of the Ethics Committee complained about the delay and the Senate's refusal to confront disciplining one of its own. But there also was an undercurrent of sympathy for Williams.

"He's like the fellow who had his car stolen because he left his key in the ignition and the window open," one senator said. "You know it wouldn't have happened if he had taken the key out and locked the doors. But you feel sorry for the guy anyway."