A growing number of Americans are using tax havens in such places as the Bahamas, the Cayman Islands and Panama as an illegal means of hiding about $20 billion a year from the Internal Revenue Service, a Senate subcommittee said yesterday.

The panel's report said that narcotics traffickers, who deal in vast amounts of illicit cash, still provide the bulk of U.S. funds laundered through foreign tax havens. But it said its "striking conclusion . . . is that offshore banking accounts are being used by people one might regard as 'average American citizens.' "

The Senate permanent subcommittee on investigations, headed by William V. Roth Jr. (R-Del.), urged the Reagan administration to impose tough sanctions -- including limits on direct airline flights -- against foreign tax havens that refuse to cooperate with U.S. law enforcement officials.

The United States should deny income tax deductions for losses and expenses incurred in any foreign country that will not negotiate access to its financial records by U.S. investigators, the panel said. Many low-tax foreign havens will not provide such records because they do not recognize tax evasion as a crime.

Banking is a booming industry in small Caribbean nations such as Anguilla, Barbados, and the Turks and Caicos Islands, which have joined such traditional havens as Switzerland and Hong Kong in attracting a flood of foreign capital by adopting strict secrecy laws. The number of banks in Anguilla, for example, jumped from two to nearly 100 in two years.

Such offshore banks have provided a repository for drug smugglers, who use private jets to carry millions of U.S. dollars at a time. But the subcommittee found similar patterns among ordinary citizens, such as a North Dakota farmer who deposited large sums in a bank in the Turks and Caicos Islands.

The report said that unscrupulous shelter promoters "exploit the anti-IRS biases of some of their fellow citizens . . . . These con men persuade susceptible individuals that they can legally avoid taxation by placing their assets in an offshore trust."

A senior IRS official said his agency is conducting 300 criminal investigations of tax shelters, half of them located abroad. He said four cases over the last year each involved more than $100 million in sheltered income.

"We're finding an increase in the blue-collar worker, the middle-income person getting into these shelters," the official said. "You can read the ads in The Wall Street Journal."

In a typical scam, he said, an investor will receive a report showing fictitious commodity losses from a brokerage house in the Cayman Islands. "In many of these situations, there are no commodities being traded," he said, "it's just paper being shuffled."

"The auditor will be given very legitimate-looking documentation. We have no way of going over to those foreign countries to find out what was bought and sold. It's an exercise in futility." He said the IRS must rely on informers and undercover agents to penetrate these shelters.

The subcommittee said the Justice Department's approach to foreign tax havens is "too fragmented" and that information is not shared among law-enforcement authorities.

"The Justice Department did not know what it had even in its own files," a subcommittee investigator said. "The Justice Department uses our list because they never compiled one." He called federal regulators "woefully ignorant on this subject," saying that U.S. bank examiners give low priority to tracking the movement of large amounts of cash.

The Treasury Department fined Crocker National Bank of San Francisco $2.25 million Tuesday for failing to report nearly $4 billion in large cash transactions, most of it suspected drug money funneled through Hong Kong banks. Sixty banks have admitted such reporting violations since February, when the Bank of Boston pleaded guilty to violating the Bank Secrecy Act. Banks have since tripled their reports of large currency transactions.

Stricter reporting would make it more difficult to hide narcotics profits, the Senate investigator said. "Drug smugglers deal in an unbelievable amount of cash, most of it in small bills," he said. "They've got to launder it, and the easiest and best way to launder it is to take it offshore."

Panama has become a burgeoning financial center. In 1982, the report said, more than $1 billion was funneled through a large Panama bank to Federal Reserve banks in New York and Miami, or four times the amount of such transactions in 1980.

A major reason for this boom is a Panama law allowing secret registered companies. "Anyone holding dirty money in Panama usually has a registered company," the report said. " . . . A registered company can be available within hours. One who walks into the right law firm in Panama City will find them already formed and 'on the shelf' . . . .

"There are no requirements of any kind to be met by an applicant for a registered company. Nor do application procedures assure that actual owners of companies are identified," the report said.