The Senate Ethics Committee yesterday voted to file formal charges against Sen. Harrison A. Williams Jr. (D-N.J.) because of his Abscam bribery conviction. The action could lead to the first expulsion in the Senate since the Civil War.

Chairman Malcolm Wallop (R-Wyo.) and Vice Chairman Howell T. Heflin (D-Ala.) told reporters that the panel based its 6-to-0 vote to hold disciplinary hearings on the trial evidence and the jury's decision.

Williams and a co-defendent were convicted Friday on nine counts of conspiracy, bribery and related charges for agreeing to use Williams' influence to get government contracts for titanium in return for a $100 million loan from an undercover FBI agent posing as an Arab sheik. Williams was videotaped in meetings with the "sheik" and his front men.

"There is substantial credible evidence that he broke Senate rules," Wallop said. He and Heflin said they will hire a special counsel to prepare for the disciplinary hearing.

He said the Senate will not await the outcome of Williams' court appeals, noting that the committee is judging him against Senate rules, not criminal statutes. The House expelled one member and forced two others to resign after earlier Abscam convictions.

Williams, 61, former chairman of the Senate Labor and Human Resources Committee, has not returned to Washington. But he issued a statement through his office yesterday saying that the committee's action was "entirely appropriate."

He also sent a personal letter to his Senate colleagues Monday, explaining the complicated legal issues in his case and noting a Philadelphia judge's opinion that he had done nothing wrong.

The Williams case was considered the most difficult of the Abscam investigation, which also resulted in the convictions of six House members. Videotaped evidence at his trial showed that he turned down a cash bribe offer to help the "sheik" with an immigration problem.

In another incident, undercover FBI informer Melvin Weinberg, a convicted con man, urged Williams to boast of his influence just before a key June, 1979, meeting with the "sheik."

In his statement, Williams said he welcomed the ethics panel inquiry "since I feel that there are numerous troubling issues about Abscam which should be brought to the attention of the committee as it considers my case."

He has rejected suggestions that he resign because of the conviction, although he has started saying, "I will take the appropriate action at the appropriate time."

Sen. Mack Mattingly (R-Ga.), a member of the Ethics Committee, said before yesterday's vote that Williams should be barred from voting, and that the Democratic minority should "take a lead in policing" Williams' activities.

While saying that he reserved judgement on expulsion, he said he didn't think the public "will condone a convicted felon serving in the highest elected body in the world." Mattingly was elected last year after his opponent, Sen. Herman E. Talmadge, was denounced by the Senate for financial abuses.

A member of the Senate Democratic leadership said Williams' situation was discussed at a meeting yesterday morning, but that there was no consensus to push him to resign.

His seat is up next year, and an early resignation would allow Democratic Gov. Brendan T. Byrne to fill the vacancy, and let the party have a new incumbent running for a full term. "If Pete runs, we've lost another seat," one Democratic senator said yesterday.

One Ethics Committee source said E. Barrett Prettyman Jr. was mentioned at yesterday's meeting as a possible special counsel for the Williams case. But Prettyman is now working for the House Ethics Committee on a potential disciplinary proceeding against Rep. John P. Murtha (D-Pa.), who was mentioned as a co-conspirator in one Abscam case, but was not indicted.