Fears are growing within and outside the Internal Revenue Service that budget cuts mandated by the new balanced-budget law will cripple the IRS' ability to collect the nation's tax revenue.

Although the Reagan administration plans to request more than $200 million in additional funds for the IRS for fiscal 1986, the agency will not be exempt from the first round of cuts under the so-called Gramm-Rudman-Hollings law. When the law takes effect March 1, $11.7 billion will have to be trimmed from the federal budget, including an estimated 2-to-3 percent cut for the IRS.

Unlike most other federal agencies, the IRS produces revenue, rather than only consuming it. So budget cuts at the IRS can reduce tax collections so much that they offset the savings in spending, tax experts said.

"If you treat the IRS like you treat everybody else, it costs more than you make. You don't need to be a genius to figure that out," said former commissioner Sheldon S. Cohen, who is drafting a letter on behalf of himself and five other former commissioners protesting the cuts mandated by the law. The letter will be sent to Treasury Secretary James A. Baker III and Office of Management and Budget Director James C. Miller III.

Analysts disagree on the precise ratio of dollars gained to dollars spent at the IRS, but it can be as high as 20 to 1 in certain divisions of the agency and is probably 8 to 1 or 9 to 1 for the entire IRS, experts say.

And, because the processing of returns and issuing of refunds during the tax-filing season for 1985 needs to be improved, tax collection will be hampered further under fiscal cuts. Filing was a disaster in the nation's 10 service centers last year, and officials have said they will use the $31 million that Congress added to this year's appropriation to improve return-processing.

That helps frustrated taxpayers, but could lead to a decline in tax collections in the long run, congressional and private-sector experts said.

However, Gramm-Rudman-Hollings requires cuts in each program, not just for each agency. So the IRS will have limited flexibility in improving taxpayer service if its funds are cut more, officials said.

"This bill will bankrupt the country because it . . . cripples the revenue-producing services," said Robert M. Tobias, president of the National Treasury Employees Union. The NTEU -- which is challenging the new law in court -- calculates that the March 1 cuts will cost the IRS $100 million, and will cost the U.S. Customs Service, another revenue-producing agency, $21 million.

For the IRS, that would more than wipe out the $31 million increase from President Reagan's budget request that Congress agreed on in December after almost a year of wrangling over the IRS budget for fiscal 1986, which began last Oct. 1.

The upcoming supplemental appropriation proposal, however, would give the IRS an offsetting increase. Sources placed it at between $230 million and $260 million, and said it was one of the few additional funding requests for 1986 that will be included in the president's budget for fiscal 1987, which will be proposed Feb. 3.

"They may seem inconsistent, but in fact they are both going forward," said OMB spokesman Edwin L. Dale. "It's a very rare case in which we had agreed at the end of the appropriations process to a higher level than Congress appropriated. It's an anomalous case."

Currently, the IRS is funded at an annual level of $3.64 billion. The administration first had called for cutting 1,254 jobs and an annual budget of $3.51 billion for 1986. The administration later supported an appropriation level even higher than Congress eventually agreed upon.

IRS officials point out they are more efficient than they once were, producing more revenue with less manpower.