Republican frontrunner Mitt Romney was overheard earlier this week saying he might eliminate the Housing and Urban Development Department.

President Obama’s advisers say he is trying to preserve HUD’s support for housing aid, but advocates for the poor say his budget plans would raise rents on very-low-income Americans and could cause some families to lose assistance.

In this age of austerity, whoever wins the presidential election will have to make tough decisions about where to find budget savings, and housing advocates worry that HUD could be a top target, at a time when the agency is already under stress.

The agency’s difficulty in meeting needs has grown only tougher with the onset five years ago of a national housing downturn, which pushed unemployment sharply higher. Rents also increased, as more people lost their homes to foreclosure or were shut out of buying a home and sought rental housing instead.

In 2009, 4.6 million poor people received aid while 14.3 million still needed it, according to HUD and the Center on Budget and Policy Priorities, citing the most recent statistics available.

Obama and Romney face pressure to target budget savings to tame the nation’s debt. The president and Congress agreed last summer to set historically tight caps on domestic spending. In addition, deep cuts will take effect next year unless lawmakers agree to raise taxes or find other savings. Some elements of the safety-net will be spared — Medicaid and food stamps — but not HUD.

“HUD is in a much more precarious, dangerous place than many other programs,” said Sheila Crowley, president of the National Low Income Housing Coalition.

HUD Secretary Shaun Donovan told Congress last month that the administration had been forced to make decisions it did not want to make and urged lawmakers to increase revenues to avoid burdening the poorest families even more.

“Protecting current families required us to make choices we would not have made in a different environment,” he said.

Still, there are profound differences between how Obama and Romney are likely to deal with the budget pressures.

Romney may seek to abolish HUD, and though he hasn’t given many details, he has signaled support for the domestic spending approach embraced by House Republicans. That approach dramatically reduces spending on programs such as housing assistance, which would likely mean that far fewer low-income people have access to rental aid.

“Gov. Romney is committed to finding areas in the federal government where he can increase efficiency and reduce spending. He is also interested in returning to the states responsibility and resources for programs that they can more efficiently and effectively administer,” Romney spokeswoman Andrea Saul said.

When Romney was overheard by reporters at a private fundraiser saying he’d eliminate HUD, Obama’s campaign pounced. “In order to fund his $5 trillion tax cuts for the wealthiest Americans, he would make deep cuts in programs essential to the middle class like . . . housing,” spokesman Ben LaBolt said in a statement.

Obama’s approach is to try to keep providing aid to everyone currently receiving assistance, officials say, avoiding cuts in rental aid next year. But housing advocates say the approach is inadequate, in part because it would require poor families to come up with more rent money.

The Center on Budget and Policy Priorities, which is close to the White House, said the president’s plan has features that are “harmful to low-income families.” The National Low Income Housing Coalition said it relies on “lowest income Americans to bear part of the burden of deficit reduction.”

Advocates of additional rental aid say the president tried to protect the housing safety net. But they say it doesn’t provide enough money to continue aid to all people whose rent is subsidized by the government. As many as 55,000 low-income families could lose aid, according to CBPP. The White House disputes this conclusion.

Part of the way the administration tries to make up for the funding gap is by increasing the minimum monthly rent that must be paid by recipients of aid to $75. But critics say that could require $25 or $50 more in monthly rental costs, a huge burden on people earning just a few hundred dollars a month.

“The great majority of them would see a rent increase of 50 percent or more per month,” said Douglas Rice, senior policy analyst at CBPP. “It’s increased hardship for families. It’s less money for food and transportation.”

Adrianne Todman, executive director of the D.C. Housing Authority, said the agency houses more than 50,000 District residents with funds that overwhelmingly come from the federal government. She said any further cuts in funding would be “detrimental to the folks that we serve,” particularly because organizations like hers have been wrestling with growing demand and shrinking resources for years.

“We’ve learned to adapt,” Todman said. “But you can only adapt so much, then you begin to have to shut down services and shut down your capacity to be a landlord.”

Todman said she has no doubt that leaders at HUD, in the Obama administration and many lawmakers on Capitol Hill care about the mission of public housing. Still, she said, “The issue that they face is the economic situation. Right now, there’s a push to preserve what we have.”