The Internal Revenue Service plans to audit tax returns of Sen. Herman E. Talmadge (D-Ga.) because of alleged irregularities in his financial affairs, informed sources said yesterday.

Talmadge, a 22-year Senate veteran, is under investigation by the Senate Ethics Committee in connection with allegations that he received money from constituents without reporting it and used an office "special account" that included political contributions to pay for some personal expenses.

Talmadge has damaged to comment on newspaper reports about his finances, saying he would defend himself only to the ethics panel.

An aide to Tal madge said the senator had received a refund the last time he was audited. "I hope I get one this time." Talmadge said according to his press spokesman, Gordon Roberts.

According to Talmadge's accountant, however, Talmadge was audited in 1976 and was assessed taxes on $5,907 in unreported income. It could not be learned whether the IRS audited Talmadge last year.

Sources said yesterday that the IRS would hold off its investigation into Talmadge's finances until the Senate panel completes its probe.

An IRS spokesman said yesterday, "We're interested, of course, in anything that has been reported about Sen. Talmadge." He declined, however, to comment on whether the IRS had made plans to investigate Talmadge's finances.

However, other knowledgeable sources said the IRS would begin an audit of Talmadge's financial papers stretching back to at least 1975, the earliest year covered by the IRS statute of limitations. The probe will not begin, they said, until completion of the Ethics Committee's investigation.

In addition to allegations relating to a "special account," there have been reports that Talmadge did not list on his tax returns thousands of dollars in birthday gifts from supporters over the years and that he failed to pay federal gift taxes on thousands of dollars in gifts to his former wife, Betty.