The Senate Select Committee on Ethics formally charged Sen. Herman E. Talamdge yesterday with five "possible violations" of Senate rules, each involving allegations of financil worongdoning by the powerful Georgia Deomcrat.

At the smat ime, a committee spokesman said an investigation into the financial affairs of Sen. Edward W. Brooke (R-Mass.) would probably be dropped because Brooke lost his Senate seat last month and thus the committee will lose jurisdiction over him.

The charges against Talmadge were outlined in a one-page letter dated Monday and signed by committee chairman Adlai E. Stevenson (D-Ill.) and Sen. Harrison Schmitt (R.N.M.), the vice chairman. Stevenson had told Talmadge of the substance of the charges against him in a telephone call Monday after the committee voted 14 to 1 to initiate a rare trial-like hearing of Talmadge by his Senate colleagues.

According to the letter, the possible violations by Talmadge include:

Submitting vouchers to the Senate claiming reimbursements due for nonexistent expenses.

Filing false campaign expense report with the secretary of the Senate.

Incorrectly reporting taxable gifts to his former wife, Betty.

Failing to report gifts and some property holdings to the secretary of the Senate.

Improperly converting campagin contributions for personal use.

Committee sources said yesterday that four of the five charges could constitue violations of the law. Failure to report gifts and property to the secretary of the Senate is a violation only of Senate rules, they said.

The letter also noted that there was "no substantial credible evidence" to conclude that Talmadge had acted incorrectly or violated Senate rules in his real-estate transactions in Georgia. Newspaper reports had indicated that Talmadge may have used his influence as a senator to profit from at least one such transaction along the interstate highway system in his home state.

The committee based its charges against Talmadge on recommendations made last month by its special counsel, Carl Eardley. In the letter, Stevenson and Schmitt promised to give the full details of the allegations to Talmadge within five days.

Eardley said Monday that he expected the hearings on the allegations to last about a week. He said he did not expect them to get under way until after the Senate convents Jan. 15.

A spokesman for Talmadge said yesterday that the senator would have no comment beyond the short statement he issued after the committee vote Monday. Talmadge said then he was certain he would be cleared of "intentional wrongdoing" by the committee.

One committee member, Sen. Robert B. Morgan (D-N.C.), was reported yesterday to be quietly attempting to negotiate a settlement of the charges against Talmadge before any hearing by the ethics panel. Morgan is an old friend of the Georgia senator and the closest one to him on the ethics committee.

Talmadge's spokeman said yesterday that he was not aware of any attempt at bargaining over the charges but the Morgan did call Talmadge last week to inform him he would vote aginst the Georgian on Monday. A spokesman for Morgan said yesterday he could not comment on the plea-bargaining report.

The charge that Talmadge converted campagin funds to his personal use appears to indicate that the committee accepted the testimony of Daniel Minchew, Talmadge's former aide, over Talmadge, who denied the allegation.

Minchew has told the committee he set up a secret bank account here in 1973 while he was an aide to Talmadge and funneled through it $26,000 in reported and unreported campaign contributions. He said Talmadge approved of the account. Talmadge has countercharged that the account was an embezzlement scheme by Minchew and that he was unawareof the account and did not know where the money went.

The charge could place the committee in conflict with a federal grand jury here that is also investigating the secret bank account. Eardley and others have said Talmadge does not appear to be the target of the federal investigation.

A committee spokesman said yesterday the investigation of Brooke would probably terminate with a report to the committee next month on investigators' findings. She called on Brooke "highly unlikely" no matter what the report finds. Brooke's spokesman had no comment yesterday.