The Internal Revenue Service has filed suit against former Rep. Edward A. Garmatz (D-Md.), claiming he owes federal income taxes on more than $11,000 given to him at a testimonial dinner after he decided to retire from Congress in 1972.

Most of the money was collected from influential members of the maritime industry before Garmatz, then head of the House Merchant Marine and Fisheries Committee, decided to step down.

Garmatz said he agreed in 1975, when his tax returns were being audited, to pay taxes on $3,000 in campaign contributions. But in his response to the IRS civil suit, which was filed in September, the former congressman contends that the remainder of the money he received at the dinner, $11,422, represented a gift and was therefore not taxable as income.

"I paid the income tax man (in 1975), and I thought he said everything was settled . . . They told me so," Garmatz said yesterday in a telephone interview from his office at the Maritime Institute of Technology and Graduate Studies, where he works as a consultant.

Garmatz refused to explain why he treated some of the funds collected at the dinner as taxable income while contending that the remainder was a nontaxable gift. He referred this and all other questions to his attorney. Arnold M. Weiner.

In his reply to the IRS suit, Weiner contended that, after Garmatz decided to retire, his campaign committee called several of the contributors and asked what should be done with the money.

As a result of these conversations, the Garmatz for Congress Reception Committee decided to hold a dinner in the congressman's honor and give him a check for any money left over after the event was paid for, Weiner's reply said.

Weinder also represented Garmatz at the time the former congressman was indicted by a federal grand jury on charges of conspiring to take $15,000 in bribes from the officers of two East Coast shipping firms. That indictment was dismissed in January 1978 at the request of Deputy U.S. Attorney Russell T. Baker after prosecutors discovered that a key witness had lied.

Baker, who is now Baltimore's U.S. attorney, said yesterday that he was unfamiliar with the recent IRS action.

Herbert Seidman, who supervises the IRS attorneys handling the case, said he was not familiar with the earlier criminal action against Garmatz.

Another familiar with both cases said they were unrelated.

Garmatz, now 76, said yesterday that the new IRS contentions came as a surprise to him when he found out about them earlier this year.

In earlier tax cases involving political figures, courts have determined that campaign contributions converted to personnal use by candidates are taxable as income, according to an attorney familiar with the laws involved in the Garmatz case.

However, people who receive gifts normally are not required to pay taxes on them.

Since the money in dispute was turned over as a gift by the officials of his onetime campaign committee, Garmatz claims he shouldn't have to pay tax on it.

In its suit, the IRS is asking Garmatz to pay an additional $5,180 in taxes.