The Senate Select Committee on Ethics opens hearings today into the case of Sen. Herman E. Talmadge, marking the first time in a dozen years that the Senate has reached the so-called "trial" stage of proceedings against one of its own.

For the 65-year-old Democrat, a 23-year Senate veteran, the proceedings could be a crucial factor in determining whether one of Georgia's oldest and most revered political dynasties will come to an end.

The case against him centers on five charges: filing false expense claims with the Senate, converting campaign gifts to his personal use, filing false campaign disclosure reports, omitting gifts and property from his Senate disclosure reports, and failing to report gifts to his former wife on his income tax.

Committee members say they also will raise questions about a secret bank account maintained here in Talmadge's name, and about a cache of $100 bills that his former wife, Betty, told the committee this month was kept until 1974 in an old overcoat in their apartment here.

Much of the information against Talmadge has been supplied by the senator's former chief administrative aide, Daniel Minchew. Minchew, 39, who was once close to Talmadge, has supplied the ethics panel with key material and testimony indicating that Talmadge profited from illegal sources of funds over the years.

Talmadge has retaliated by labeling Minchew an "embezzler" and blaming discrepancies in his office accounting on sloppy bookkeeping by aides. He has not spoken out ont he allegation by his former wife of the cache of $100 bills.

Leadoff witness in the ethics committee hearing will be T. Rogers Wade, Minchew's successor as the senator's administrative assistant and his campaign chairman. Talmadge not only has refused to be the opening witness, but has not told the committee whether he plans to appear at all.

The job investigating Talmadge was not eagerly sought by his colleagues. When the panel was organizing this month, one new member was picked only moments before a deadline. The selection Quentin Burdick (D-N.D.), made no secret of his unhappiness.

A second new member, Jesse A. Helms (R-N.C.), was a compromise after Talmadge allies objected to the original Republican choice. Helms is the ranking minority member on the Senate Agriculture Committee, which Talmadge heads.

The ethics committee also includes Chairman Adlai E. Stevenson (D-Ill.), Vice Chairman Harrison Schmitt (R-N.M.), and Sens. Robert Morgan (D-N.C.) and Mark Hatfield (R-Ore.).

The choice of punishment if Talmadge is judged guilty is up to the entire Senate. The most severe would be Senate expulsion, an unlikely possibility. The ethics committee indicated in a plea-barganing offer to Talmadge last week that its choice is a formal censure, which would involve appearing before the Senate to accept public chastisement and possibly other penalties. Talmadge has not replied to the committee's offer.

The ethics panel's deliberations will be closely watched in Georgia. The senator's term expires next year, and potential candidates are lining up to run if he falters. They include several congressmen and Atlanta Mayor Maynard Jackson. Gov. George Busbee has denied that he will run against Talmadge, but a recent Atlanta Constitution poll put him way ahead of Talmadge if he does.

Talmadge has vowed to run again. Some political observers in Georgia believe he will carry out that pledge unless, as one put it last week, "he comes out of this with blood dripping all across the floor." CAPTION: Picture, Former aide Daniel Minchew says his boss profited from illegal sources of funds; Sen. Talmadge calls Minchew "embezzler," says office ledgers were sloppy. AP