Conservative political groups are mobilizing against a key element of a bipartisan infrastructure deal, and their opposition could make it harder for the U.S. government to collect unpaid taxes.

Congressional Democrats and Republicans have agreed to increase funding for the Internal Revenue Service so that the agency can bring in more tax revenue, hoping the money can help pay down some of the infrastructure package’s expected price tag. The early contours of the infrastructure blueprint have won the White House’s support, but the IRS provision in particular is drawing opposition from well-funded conservative groups, which are strongly opposed to expanding the reach of a tax-collection agency that they long have alleged is politically motivated.

Among the conservative groups spearheading the opposition are the Committee to Unleash Prosperity, FreedomWorks, the Conservative Action Project, and the Leadership Institute. They are preparing a letter that warns Republicans should not negotiate with the White House unless they agree to “no additional funding for the Internal Revenue Service.”

The letter, obtained by The Washington Post ahead of its release, is expected to gain support from at least a dozen other conservative groups this week, with plans to send it soon to Senate Minority Leader Mitch McConnell (R-Ky.) and other Senate GOP leaders.

The Coalition to Protect American Workers, a group founded by Trump administration aide Marc Short, is hoping to raise $25 million for a campaign against the IRS enforcement plan. The group has already purchased “multiple” six-figure ad buys — including in Kansas, Pennsylvania, Georgia and Arizona — targeting Democrats and Republicans who support the measure, a spokeswoman for the group said. This fundraising goal was first reported by Axios.

The conservative advocates’ push could test their political muscle under the Biden administration. Under President Barack Obama, conservative activists connected to the tea party movement successfully pushed GOP lawmakers to fiercely oppose new government spending, leading to major cuts that hamstrung the IRS. But President Donald Trump spent aggressively while in office, blowing past the warnings of the party’s libertarian wing and deflating conservative opposition to new programs.

It is unclear whether the GOP zeal to cut federal spending will reemerge under a new Democratic administration. A successful effort could imperil bipartisan support for the $974 billion infrastructure package, a top priority of the Biden administration as it seeks major upgrades to the nation’s roads, bridges, highways and other public-works projects.

“Republicans are going to double the IRS budget? That’s crazy. There’s very strong opposition to this,” said Stephen Moore, a former outside economic adviser to Trump who is leading the effort. An op-ed on the measure that Moore wrote with Steve Forbes also has circulated in Congressional Republican offices.

The bipartisan infrastructure deal reached last month by the White House and a group of Democratic and Republican senators proposes $40 billion for heightened IRS enforcement, with an expectation that it would result in around $140 billion in new revenue. In theory, at least, the idea has broad support among Democrats and Republicans alike, who in recent years have pointed to weaker IRS enforcement and estimates of the nation’s persistent “tax gap,” or the difference between what taxes are owed to the government and what is actually paid.

“I am working with my colleagues in both parties to improve taxpayer services and the collection of taxes owed, while putting in place responsible reforms to prevent the IRS from overstepping its authority and protect taxpayer data,” said Sen. Rob Portman (R-Ohio), one of the lawmakers who helped craft the bipartisan infrastructure deal.

Over the past decade, persistent budget cuts have hurt the IRS’s ability to conduct audits, including those targeting wealthy and large corporations. Tax experts have expressed alarm that the weakening of the IRS has helped fuel the increase in U.S. income inequality, in part because the rich have more tools to dodge the increasingly weak tax collection agency.

Lawmakers who endorse a boost to the IRS as a means of financing infrastructure changes have said little publicly as to exactly how they plan to spend the money. For now, their bipartisan deal is expected to set aside some of the funds to aid hiring at the agency, improve its dated information technology and address taxpayer compliance issues, according to two Senate aides with knowledge of the plan who spoke on the condition of anonymity to describe the talks. They said their effort ultimately would include some still-unfinished guardrails to try to assuage Republican lawmakers who long have expressed skepticism about the IRS.

“Any more funding for the IRS must come along with robust oversight and accountability measures. Additional funding also can’t come at the expense of taxpayer rights or services,” Sen. Charles E. Grassley (R-Iowa) said in a statement. “I haven’t seen the details of any specific proposal. But I’m skeptical of some of the huge numbers being talked about.”

The Biden administration previously had proposed an $80 billion cash infusion to toughen IRS enforcement efforts as part of its “American Families Plan” proposal earlier this spring, as well as specific policy changes such as hiring agents to tackle complex returns, forcing firms to disclose more information and overhauling outdated technology.

Because of GOP opposition, these policy changes are not expected in the bipartisan infrastructure agreement, even though experts believe they are the most likely to make a dent in the tax gap. Instead, said Sen. Angus King (I-Maine), a member of the negotiating group, the focus will be on adding to the IRS’s enforcement staff, which declined by 30 percent between 2010 and 2018, according to the Congressional Budget Office.

“The assumption is it’s additional resources that would be directed specifically to those activities which would increase revenue,” King said.

But even these more modest moves may be a tough sell among the GOP. Senate Republicans — including some of the moderates who have indicated interest in an agreement — may oppose any deal if it includes not just more funding for the IRS but bolsters a crackdown on tax evasion, even if focused on corporations and the rich, according to multiple Congressional aides and lobbyists in close contact with lawmakers.

“Enforcement of tax laws is one thing, but what people are concerned about is aggressive audits,” said Jason Pye, a federal lobbyist formerly at FreedomWorks, a conservative organization.

Kevin Kuhlman, vice president for federal government relations at the National Federation for Independent Business, said that while his group has not taken a formal position on an IRS budget boost, proposals to increase funding should “proceed with caution.”

“Reports of increased audits, enforcement and reporting requirements raise red flags with our members,” Kuhlman said. “We would urge and encourage instead increased compliance assistance, better customer service and remedying the processing delays.”

There is some reason for negotiators to be optimistic. Senior Republicans are under intense pressure from business groups to approve an infrastructure deal that corporate America has long sought. Many Republicans also see a big bipartisan infrastructure deal as the best hope for weakening the chances of a broader Democratic spending package, which probably would have bigger tax increases and include climate and social programs that the GOP is eager to kill.

Supporters of the efforts to enhance the IRS plan to hammer home the message that those who oppose an increase in the agency’s budget are helping rich tax cheats and hurting ordinary, law-abiding people.

“Why is it fair to working Americans who pay their taxes to allow people who can afford fancy lawyers and accountants to cheat?” King said. “I’m not talking about tax avoidance. I’m talking about outright cheating, hiding income.”

Trump’s budgets also called for increasing funding for the IRS — in part because the GOP was eager to improve implementation of the 2017 tax law — which some moderate lawmakers have seen as giving political cover to Republicans who fear supporting the IRS funding increase will expose them to attacks from the right.

“If this was the dealbreaker, fundamentally the negotiations wouldn’t have gotten to this stage,” said one GOP lobbyist briefed on internal dynamics, speaking on the condition of anonymity to discuss interactions with Capitol Hill. “If this is the glue that holds the whole thing together, it’s preferable than most of the other pay-fors.”

Animating Republican opposition is more than a decade of ill will between the GOP and the IRS. Nearly a decade ago, a top IRS official acknowledged its auditors had scrutinized the finances of conservative leaning groups based on their names — raising the specter of political targeting even though the agency did eye some liberal nonprofit groups as well.

More recently, conservatives say the IRS is to blame after a major story by ProPublica revealed some of the wealthiest Americans pay relatively little in taxes. Many Republicans suspect IRS officials were behind the disclosure, although ProPublica has said it does not know where the information came from and there is no evidence the IRS staff did so. Tax experts also speculate that a lack of funding also may have made the leak more likely, either by weakening the tax agency’s IT protections or by forcing it to offload sensitive data to third-party contractors.

“The people who were against it to begin with have even more fodder, which makes the supporters more nervous … They’re fine on the substance of giving the IRS more money, but that’s maybe only five to 10 [Republicans] in the Senate,” said Donald Schneider, who worked for Republicans on the House Ways and Means Committee. “I think the coalition can hold for it — especially knowing what else is in the balance — but it’s in a precarious position now.”

The groups leading the opposition to the IRS budget increase include those that have received funds from major conservative donors, including the Mercer Family Foundation, the Sarah Scaife Foundation and Donors Trust, a donor-advised fund that gives to conservative and libertarian causes. One signatory of the letter, Phil Kerpen of American Commitment, worked for five years at Americans for Prosperity, the main political arm of the influential Koch network.

Executives at the Mercer Family Foundation, the Sarah Scaife Foundation and Donors Trust did not respond to requests for comment.

For now, the task falls to key Congressional committees to translate the policy into law. One potential compromise floated by some experts would be to increase the funding and power for the IRS Oversight Board, a long-standing focus for Portman. The bureau, created in 1998, is currently viewed as a “toothless tiger,” one GOP tax expert said, and could be bolstered as part of a broader deal. According to its website, the oversight board does not have enough Senate-confirmed members to make up a quorum and has suspended operations.

“There’s room for legislation to be written in a way to explain how that money would be divided, and the question is how specific that legislation would be,” said Janet Holtzblatt, an expert in IRS enforcement at the nonpartisan Tax Policy Center.