In the face of strong protests from congressional Democrats and public-housing authorities around the country, the Department of Housing and Urban Development has decided to back off a bit from its plan to save money by rewriting a rent-subsidy formula, according to HUD sources.

A compromise version of the new formula, which applies to existing buildings in the Section 8 program, is expected to go to Congress this week, perhaps as early as today or Tuesday.

HUD has been working since last year on the rewrite, which it said it began "in the interests of cost containment." Initially, the department proposed a version that would have saved as much as $100 million in a program that cost about $2.2 billion in fiscal 1983.

But housing-authority representatives complained loudly that the proposal would have cut the government rent ceiling to the point where many, and perhaps most, new tenants would have been unable to find a place to live. As one Vermont official put it, the new formula would be "especially cruel" because it would "effectively bar eligible applicants from participating in the program."

At issue is what is called the fair-market rent standard, which HUD uses to determine what subsidy it ought to pay in a given area. A family participating in the program puts up 30 percent of its income, and HUD pays the difference between that and the fair-market rent.

The department now bases its payment on the rent paid by the 50th percentile of families who have recently moved into their houses or apartments.

Under the initial rewrite, the department would have shifted to the 40th percentile of all rents in an area, excluding housing built in the past two years and including tenants in assisted housing.

That would have cut the fair-market rents substantially in most markets, and it touched off efforts in Congress to block the new rules.

The current law leaves the formula to the department's discretion. By last spring, HUD seemed ready to go ahead and, in effect, challenge Congress to do something about it. And as the year wore on, the chances of Hill action seemed increasingly unlikely, as the housing authorization bill bogged down in the Senate.

The House, which passed its housing authorization bill in July, included a provision to force HUD to stay with the current standard. The Senate housing bill, which has cleared committee but not the floor, would allow some reduction, but not as much as HUD had proposed.

Now, HUD sources said, the department has decided to compromise, somewhat along the lines of the Senate measure. Under this version, the fair-market rent would be pegged to the 45th percentile of rents paid by families who have moved recently. New housing still would be excluded, but so would rents charged in public housing.

The fair-market rent still would be lowered quite a bit, and organizations that represent poor tenants are still unhappy, as are many housing-authority officials who feel that even the current standard is too low.

HUD officials last week were not prepared to say how much money the compromise standard would save. But one source said the new fair-market rent formula, coupled with some "definitional tinkering" with the percentile calculation, will retain "better than half" the savings envisioned in the initial proposal.

With Congress scheduled to recess in October, HUD needs to get the new regulations to Congress soon for a 30-day review period if they are to be implemented by winter. Asked if that review is mandatory despite the Supreme Court's decision striking down the legislative veto, one department official said, "I don't know, but we keep sending them up there."