Americans in greater numbers are finding new ways to cheat on their taxes, according to reports to Congress by the Treasury Department and the Internal Revenue Service.

The two agencies said tax-shelter fraud is increasing rapidly, taxpayers are finding new ways to circumvent the compliance rules in the 1982 tax act, and voluntary tax-law compliance by individuals is decreasing.

In two days of hearings before the Senate Finance committee and its Internal Revenue subcommittee, Treasury and IRS officials warned that tax-shelter abuse threatens to undermine public confidence in the tax system and encourage more people to evade their tax obligations.

"A large number of the abusive tax shelters being sold today are not really tax shelters in any traditional sense, but are frauds euphemistically referred to as tax shelters," acting IRS Commissioner Philip E. Coates told the subcommittee yesterday.

Coates and John E. Chapoton, assistant Treasury secretary for tax policy, asked for legislation requiring marketers of tax shelters to give IRS the names and addresses of participants, so that IRS could notify them in advance of its intention to challenge any deduction it considers fraudulent and track down other participants when it finds one questionable deduction.

Tax shelters are investments, usually for upper-bracket taxpayers, that provide short-term losses or depreciation deductions to offset tax liability for other income. According to the Joint Committee on Taxation, Americans invested $9 billion in tax shelters in 1982 and will invest an estimated $11 billion in 1983.

The Reagan administration has argued that the tax cuts enacted by Congress, particularly those for the wealthy, would cut down on the use of tax shelters.

Most of these investments are legal and are packaged by legitimate brokers, federal officials said. E. F. Hutton Co. claims to be the largest, with more than $2 billion invested for clients since 1970.

According to Coates and Robert G. Woodward, Treasury's tax legislative counsel, however, IRS crackdowns have not halted the proliferation of illegal shelters, which Coates said are becoming "more and more egregious."

The 1982 tax act gave IRS for the first time power to sue tax-shelter packagers if IRS believes the shelter being offered is "abusive," and the agency has begun going into court, Coates said.

Coates said 16,300 tax shelter cases are pending in U.S. tax court, about a third of the court's total docket.

Woodward said the Treasury wants to clamp down on tax abuses not out of a "desire for additional revenue" but out of concern that they breed contempt for the tax laws. "People who have tax shelters tend to talk about them," he said, encouraging other taxpayers to feel that they, too, should be able to shield their income from taxation.

Unreported legal income leaped from $93.9 billion in 1973 to $249.7 billion in 1981, Coates said. Given the sustained inflation that occurred in those years, he said, the numbers are not as bad as they appear at first glance, but there also has been a small, steady decline in the percentage of income voluntarily reported. It is now down to 89.3 percent.

In the hearings, the Treasury and other IRS officials made these other points about tax compliance:

* The insurance industry has created a new kind of policy offering protection against the results of a tax-return audit. This kind of insurance, introduced last month by Victor O. Schinnerer & Co. of Washington, "permits a taxpayer to avoid liability for taxes owed by providing reimbursement where noncompliance is detected and has no discernible social benefit and can only result in increased contempt for the tax laws," Chapoton said.

* Investors are evading restrictions on "straddle" investments in commodity futures by channeling them through overseas corporations. Instead of offsetting positions in commodities, the taxpayer has a commodities loss balanced by stock holdings in a foreign company, taxed on a different basis.

* A quirk in the law allows some mutual fund investors to have a long-term capital gain and a short-term loss on the same shares, just by holding them for 31 days. Congress is likely to extend this period to six months or more.