The data breach at the IRS that left the personal information of 104,000 taxpayers in the hands of thieves is the latest wrinkle in a mammoth problem faced by tax authorities: Identity theft and its crippling consequences.

An unprecedented surge in online tax scams by increasingly sophisticated criminals has challenged the IRS to respond quickly to get ahead of the fraudsters, especially during this year’s tax season after hackers targeted TurboTax, the country’s largest online filing service.

The vulnerability of taxpayers’ personal data was identified last fall by the IRS’s independent watchdog as the agency’s number one problem. Tax officials estimate that the government has lost billions of dollars in recent years to fraudulent refunds filed by hackers who steal personal information on tax returns, then use it to claim a refund in a taxpayer’s name before he or she files.

This is precisely what happened in the scam announced Tuesday: Hackers used information stolen from previous breaches — including Social Security numbers, birth dates, street addresses and passwords — to complete a complex authentication process and request tax returns and other filings.

The criminals succeeded: IRS Commissioner John Koskinen called the scam a “modified form of identity theft” and said the fraudulent returns were able to claim $50 million in refunds before the agency’s IT staff detected the scheme.

The IRS had already gone into fight mode this tax season with an aggressive strategy of prevention, detection and assistance to victims. Besides installing stronger security systems, computers began flagging anything suspicious in a taxpayer’s return, from addresses that didn’t match up with what the government had on file to large deductions for self-employed people.

Many of these “suspicious” returns turned out to be perfectly legitimate. But the fight against identity theft has created big new demands on the agency and taxpayers at a time when the IRS is stretched thin by budget cuts.

In Dallas a few weeks before tax filing day last month for example, the taxpayer assistance center downtown was overwhelmed with customers who had waited hours to see a specialist. A majority of the customers who finally sat face-to-face with these staffers were carrying letters telling them they might be victims of identity theft. They needed to triple check their addresses, income, Social Security numbers and other identifying information to eliminate the possibility. It took a long time.

The efforts to fight identity theft are paying some dividends. Koskinen said the IRS assisted 875,000 victims in the fiscal year ended Sept. 30. In fiscal year 2014, 1,063 identity theft-related investigations were initiated and criminal enforcement efforts resulted in 748 sentencings, compared to 438 a year earlier, according to agency statistics.

A report issued Thursday by the Treasury Inspector General for Tax Administration found that the IRS is getting better at preventing identity theft-related tax fraud. But investigators found limits to the agency’s ability to stem the problem. The IRS does not have access to third-party income and withholding information until well after tax return filing begins, they found. The IRS is urging Congress to pass legislation to accelerate and expand its access to this data.

In the breach revealed this week, the IRS said it will contact the 104,000 taxpayers whose information was compromised, as well as the 100,000 for whom attempts were unsuccessful. The first group will be offered credit monitoring, while the second will be warned that thieves have their personal information. But victims of tax identity fraud have complained about delayed refunds and bureaucratic hassles to set their financies straight:

The problem is compounded by politics. House Republicans have slashed the IRS budget by billions of dollars in the past five years, and the cuts include resources to fight hackers. Koskinen is asking Congress for an additional $82 million in the coming fiscal year to prevent identity theft and refund fraud, much of it for long-delayed technology investments.

So far, he hasn’t had much luck convincing House Republicans to give him more resources.