Internal Revenue Service officials said yesterday that the agency is making progress in a campaign waged against tax protesters -- those who object to paying their taxes on other than pecuniary grounds.

As a result of the campaign, the IRS said, more people are paying more in penalties; more of those convicted are going to prison, and the sentences are longer.

"I don't want to say that we have won the battle, but we have scored some significant victories," said Philip E. Coates, associate commissioner for operations.

The IRS has reduced the number of protester returns in its backlog from 36,712 in 1983 to 28,324 in 1984, after a steady rise in the previous four years. The agency also is assessing more penalties for so-called "frivolous" returns and is seeing fewer claims for excessive exemptions, which result in little or no tax being withheld from a paycheck. Last year, the IRS closed 23,000 protest cases, resulting in an average collection, including tax and penalties, of more than $7,000.

However, officials said, those who once challenged the law by openly protesting paying taxes may be switching to protesting by using questionable tax shelters.

"The line between a tax shelter and a tax protester is becoming somewhat fuzzy," said Richard Wassenaar, associate commissioner for criminal investigations.

Among the new weapons available to the agency as a result of 1982 changes in tax law is a $500 penalty for filing a frivolous return, written deliberately so that it can't be processed. In fiscal 1984, 8,210 of those penalties were imposed. The courts also impose penalties of up to $5,000 on those who use the process just to delay paying their taxes. When that's added to existing penalties for fraud, "the stakes are enormously high," said Chief Counsel Fred T. Goldberg Jr.

Overt protesting became a nationwide problem during the 1970s, as groups organized with the purposes of inducing and helping members not to pay taxes. They claim a variety of reasons, Coates said, including that the income tax is unconstitutional.