Government action to fight offshore tax evasion is making it tougher for the well-off to dodge the IRS. (Andrew Harrer/Bloomberg)

Wealthy Americans used to have no shortage of places to hide their assets from tax collectors — Cayman Islands, Switzerland or Luxembourg.

But that is starting to change.

Government efforts to combat offshore tax evasion are making it harder for the well-off to dodge the Internal Revenue Service. Earlier this week, the Justice Department struck a plea agreement with Credit Suisse to turn over account information that federal prosecutors say will help them track down tens of thousands of U.S. tax cheats. The Swiss banking giant pleaded guilty to conspiring with its American clients to dupe the IRS, agreeing to $2.6 billion in fines.

And federal prosecutors say this is just the beginning.

“Our work in the offshore area is far from done,” Deputy Attorney General James Cole told reporters at a news conference announcing the Credit Suisse deal. “We expect additional public actions in this area in the coming months.”

But in the government’s push to plug holes in the tax system, critics worry that some solutions are being overlooked and some well-meaning taxpayers are being harmed.

Finding individuals who shirked their tax bill has always been difficult. Switzerland, whose centuries-old culture of banking secrecy has made it a sanctuary for the world’s rich, has waved off U.S. pleas for cooperation for decades.

Yet Swiss authorities have softened their position. In August, they reached a deal with the United States to allow some Swiss banks to pay fines to avoid or defer prosecution over tax evasion by wealthy American customers. The deal has attracted 106 Swiss banks, which have agreed to disclose information about their clients.

While the Swiss are still withholding names, Justice officials said they can use the data they are getting to track Americans who had accounts or moved money around once they learned of prosecutors’ investigations. The department stands to gather even more account information from the ongoing investigations into 13 other European banks, including HSBC and Julius Baer.

A key turning point for Justice came in 2009 when UBS turned over information about 4,700 account holders as part of a $780 million settlement with the department. Analysts say the agreement put foreign banks on notice that the U.S. government was serious about cleaning up tax evasion.

“Ten years ago, if you had asked somebody if the IRS could find out about their secret Swiss bank account, they’d say no,” said Rebecca Wilkins, senior counsel at Citizens for Tax Justice. “But the UBS case broke it all open, and now people know that the DOJ and IRS are pursuing these cases, and bank secrecy may soon be a thing of the past.”

Essential to the effort has also been the Foreign Account Tax Compliance Act, which made it easier to get information about the foreign bank accounts of American taxpayers. The law requires foreign banks to annually disclose the accounts of their American customers or pay a 30 percent tax on their U.S. investment income. Nearly 70 countries and territories, including Switzerland, the Cayman Islands and Luxembourg, have inked agreements with the Treasury Department to comply with the law. Starting July 1, countries that have signed the agreements will either collect account information or have their banks send it directly to the IRS.

“We’ve made tremendous progress,” said Itai Grinberg, associate professor of law at Georgetown University. “The combination of aggressive DOJ enforcement and legislation that has fundamentally altered the landscape of international bank cooperation . . . it’s really making a big difference.”

But there are loopholes in the tax compliance law that, among other things, allow foreign financial firms to protect U.S. offshore shell companies. And while the law targets people willfully concealing their assets from tax collectors, it has become onerous for some Americans working overseas who are paying their taxes, said Marylouise Serrato, executive director of the nonprofit American Citizens Abroad.

“We are hearing and seeing Americans being denied mortgages, bank accounts and pension funds overseas by some foreign financial institutions that are nervous about the risks,” she said.

Serrato also takes issue with one of the government’s most successful tax evasion initiatives, the Overseas Voluntary Disclosure Program. Since the program took effect in 2009, more than 43,000 taxpayers have told the IRS about their offshore accounts and paid more than $6 billion in back taxes, interest and penalties.

But Serrato argues that many of those taxpayers caught by that IRS program were not deliberately evading taxes. She has received complaints from Americans living overseas who did not know they had to declare foreign pension plans and were hit with thousands of dollars in fines once they alerted the IRS.

Wilkins of Citizens for Tax Justice sympathizes with Americans who may have made honest mistakes in declaring their overseas accounts, but she suspects they are a minority.

A bigger problem with the disclosure program, she said, is that the information collected about professionals who helped Americans set up offshore accounts is not being used to its full potential. Wilkins said federal prosecutors should more aggressively pursue the accountants and lawyers involved in the process, not just the bankers.

“They have been slow to go after those folks in the U.S., but if they did that and put some high-profile people in prison, they could shut this down pretty quick,” she said.