Sen. Herman E. Talmadge (D-Ga.) suffered two setbacks yesterday when the Senate Select Committee on Ethics rejected his plea to bar critical evidence from a hearing on his alleged financial misconduct and to adopt stricter rules of evidence in his case.

But the committee granted Talmadge attorney James Hamilton a one-week delay on the start of the hearing, moving it forward from next Monday to April 30. Hamilton argued last week that he needed a two-week delay because he had failed to receive material subpoenaed from Daniel Minchew, a former Talmadge aide and a key witness against the Georgia Democrat.

The two motions rejected by the Senate committee could have made it more difficult for the panel to conclude that Talmadge violated Senate rules in the handling of his finances.

Attorneys for Talmadge argued that material critical to the prosecution of Talmadge was not admissible at the hearing because it was removed from the senator's private files by Minchew with permission. A smiliar motion was upheld in 1967 during proceedings against Sen. Thomas Dodd (D-Conn.). Dodd was the only other senator to face a proceeding in the Senate similar to the one against Talmadge.

The committee left open, however, additional arguments for exclusion of the documents during the hearing itself.

In the second motion, the attorneys for Talmadge called for the Senate panel to impose a standard of proof "beyong a reasonable doubt" against the Georgian before the case is delivered to the Senate for final action. The standard is regularly used in criminal trials in the courtroom.

Carl Eardley, special counsel for the committee, had argued for a less stringent standard of "clear and convincing evidence" against Talmadge as the basis for forwarding the case to the Senate.

The Senate committee rejected the two Talmadge motions in single-paragraph decisions without any additional comments. It did not mention three other motions made by his attorneys during a committee session last week.

Meanwhile, sources said yesterday that the committee learned as far back as last fall from Minchew about a cache of $100 bills available to Talmadge, despite claims by Talmade that he spent almost nothing on his personal affairs. Talmadge told reporters and the committee that his day-to-day expenses were covered by $5 and $10 donations from friends and supporters in Georgia.

Earlier this month, however, Talmadge's former wife, Betty, turned over 77 $100 bills that she said were part of a cache of $100 bills kept by the pair until 1974 in their Washington apartment. The information about the money was initiated this month by Betty Talmadge, according to the sources.

The sources said yesterday that federal investigators have conducted a "line-by-line" audit of Minchew's finances and have traced about $10,000 from the secret account in the Riggs National Bank to Minchew. Last summer Minchew told investigators he got about $7,000 from the account as a reimbursement for expenses he incurred on behalf of Talmadge.

The sources said yesterday that on three sperate occasions in 1974 Minchew withdrew a total of $7,900 from the account. Nearly $3,000 of the money went into Minchew's own account, the sources said, and most of the rest was used by Minchew against a personal note held by the Riggs bank.

Minchew has also told investigators that in 1973 and 1974 he put a total of $39,000 into the bank account from illegal Senate reimbursements and campaign contributions to Talmadge. The account was maintained, he said, with the senator's knowledge.

Nearly $5,000 was taken from the account in 1973 and redeposited the next year in a legitimate Talmadge campaign account, Minchew has testified. But the sources said yesterday that except for that money and the $10,000 that Minchew withdrew for himself, the federal auditors have been unable to trace the remaining $24,000 in the account.

Another source close to Talmadge said yesterday that the senator's attorneys have been conducting their own investigation into the finances of the former aide and will seek to show during the hearing that Minchew got more than $10,000 from the account. Talmadge has denied knowing about the account or benefiting from it.