That's a lot of money — and we're all paying for it, Zucman said.
When the wealthy pay less in taxes, the rest of the population bears the burden. Either the government spends less money, providing fewer public services, or ordinary citizens pay more to make up the cost.
"The taxes that are evaded at the top have to be compensated by higher taxes for the middle class," Zucman said.
Alternatively, the government borrows the money, and interest rates increase as ordinary people have to compete with the government to get loans.
To get an idea of how much money is at stake, here is a list of a few things Congress could do with $124 billion a year.
1. Feed the poor
The money that Americans avoid paying in taxes by shifting their wealth abroad would be more than enough to feed tens of millions of people for a year.
The number of Americans receiving food stamps increased from 26 million in 2007 to 48 million in 2013. While that number has decreased as the economy has improved — falling to 45 million at the end of last year, according to federal data — helping all these people put food on the table is a huge expense all the same, and this expansion has been controversial.
Even in 2013, though, the cost of the Supplemental Nutrition Assistance Program — the technical term for food stamps — totaled less than $80 billion.
2. Pay the troops
In fact, all the money at stake in international tax shifting would be enough to pay every uniformed member of the U.S. armed forces. Excluding housing and health care, the Pentagon's personnel costs totaled $116 billion in 2014, according to the Congressional Budget Office, which is less than Zucman's estimate of $124 billion. (That's counting retirement pay as well as compensation for reservists and the National Guard. Housing was another $19 billion, and the Department of Veterans Affairs is asking Congress for $182 billion this year.)
Another source of controversy in the federal budget has been the cost of the Pentagon's F-35, a new warplane that military leaders hope will be more versatile and resilient than past aircraft. While the F-35 is intended to replace several planes currently in use, the Government Accountability Office pointed out in a recent report that the cost of the program will be twice that of four legacy aircraft combined.
For all that, the annual costs of the F-35 are projected to reach only $20 billion a year. If wealthy Americans and multinational corporations couldn't avoid paying taxes by shifting their income overseas, the Pentagon could pay for its F-35s six times over.
3. Send every kid to preschool
With just $90 billion a year, Congress could set up a national network of high-quality early-education programs open to all families, according to a recent analysis from economist Josh Bivens and his colleagues at the liberal Economic Policy Institute.
Their goal is to allow every family in the country to provide preschool and child care for infants and toddlers 4 and younger for no more than 10 percent of their incomes. The federal government would pick up the rest of the tab. The plan also calls for a staff of nurses to coach pregnant mothers and families with infants on child rearing.
Bivens, it's worth noting, says the plan would pay for itself over the long term, and that the government wouldn't need a windfall from money squirreled away in the Caribbean to make it happen. "The kids who grow up 10 to 20 years from now would be more likely to earn higher wages and avoid contact with the criminal justice system," he recently told Wonkblog.
4. Give working families cash
Sen. Marco Rubio (R-Fla.) has proposed giving families an annual tax credit of $2,500 per child to help cover the cost of raising children, to replace the current credit of $1,000. Under Rubio's plan, the credit would be worth less for affluent families with more than $300,000 a year in joint income. It would reduce the taxes that families pay on their salaries and wages, so families that pay less than a certain amount in taxes because they are poorer would also receive less in credits.
The nonpartisan Tax Policy Center projects that Rubio's credit would cost $122 billion on average over the next decade. If Americans stopped avoiding taxes, Congress would have enough money to give parents a few grand every year while still having a couple of billions to spare.
Michael Strain, an economist at the conservative American Enterprise Institute, argued that policymakers shouldn't forget about workers who don't have children. President Obama and Rep. Paul Ryan (R-Wis.), the speaker of the House, have indicated that they'd like to expand a crucial bonus for this group of taxpayers known as the Earned Income Tax Credit. The cost of the plan they've discussed would only be about $6 billion a year.
5. Borrow less money
"I would use some of this to expand the Earned Income Tax Credit," Strain said. "I would use a big chunk of it to close the deficit."
The $124 billion in unpaid taxes would be enough to eliminate nearly a quarter of the annual deficit — the difference between what the government collects in taxes and the amount it spends, which the Congressional Budget Office projects will be $534 billion this year. That's money the government borrows.
Much of that deficit is due to Social Security's shortfall. The program's actuaries project that the New Deal program will no longer be able to pay retirees all of what they're owed beginning in about 20 years, when the trust fund will be exhausted and the program will be able to pay out no more than what it collects in taxes every month.
Strain's colleague Andrew Biggs estimates that making up Social Security's actuarial deficit would cost about $182 billion a year beginning now. The unpaid taxes could go a long way toward making up that money.
6. Roads, bridges and railways
Jared Bernstein, who served as Vice President Biden's chief economist, did not agree that the money should be used to reduce the deficit. In a note to Wonkblog on Friday, he wrote that the government should work to pay down the national debt, but that now is not the time. The U.S. economy is still operating below its potential, he contends, and the goal should be stimulating the economy rather than saving money.
Bernstein wrote that the federal government should put more money toward maintaining and building the country's physical infrastructure. That includes highways, railroads, bridges, dams, levees, power lines and airports, but also public schools. Bernstein pointed to recent research that suggests the country should be spending an additional $46 billion a year or so keeping up the schools.
Zucman's estimate of $124 billion wouldn't be enough to fund the most ambitious plans for infrastructure, though. Bernie Sanders, the Democratic presidential candidate, has introduced legislation that would appropriate $200 billion a year for a period of five years. The centrist advocates at Third Way say the country needs to spend even more than that — about $240 billion a year over five years.