Five years ago, housing the poor was high on the list of national priorities, the fastest-growing federal subsidy program of the decade. The Department of Housing and Urban Development had $31.7 billion to spend on housing assistance in the fiscal year beginning Oct. 1, 1978.

But little more than a year later, with the election of Ronald Reagan, all that began to change.

The Reagan administration cut housing assistance funds by 60 percent between 1981 and 1985, from $32 billion to $10 billion, the biggest reduction in any single domestic spending program.

A $15 billion HUD appropriations bill passed last week and sent to the president is $8 billion more than the administration asked for, because Congress has consistently given HUD more money than it has asked for.

HUD requested only $2.3 billion for housing aid, and says it does not want to build any new assisted housing units at all. The agency proposed deep cuts in public housing operating subsidies and other programs, while freezing rental assistance payments at their present level for two years.

Administration officials say, however, that more poor Americans are receiving aid than ever before at the cost of little more than a third as much money spent as was spent in fiscal year 1981, the last budget of the Carter administration. Reagan officials say they have done it by dumping unreasonably expensive programs, relying on housing that is available in most communities and rehabilitating previously uninhabitable public housing units. More than 4 million people are now getting housing assistance, a rise of 800,000 since 1981, according to a HUD spokeswoman.

Despite Congress' efforts to keep pouring money into HUD's housing budgets, the agency has sometimes refused to spend what Congress appropriated, and has canceled commitments for more than 265,000 low-income housing units authorized by previous administrations, according to a report prepared for Democratic members of the Senate housing and urban affairs subcommittee.

The big cutbacks in housing programs have given birth to a theory among critics and low-income housing advocates that the administration is intent on scrapping housing aid and dismantling HUD, a charge Reagan officials dispute.

While assistance has been cut back, the number of Americans in need of housing aid has increased sharply and housing quality is declining, according to several sources. The National Low-Income Housing Coalition estimates that for every very-low-income family in subsidized housing, three others, or 7.5 million, need it but can't get it, and homelessness is increasing dramatically.

With a relative handful of new units authorized for construction, the numbers of those in need will grow swiftly, the organization says. Moreover, housing quality is declining and an estimated 500,000 low-rent units are disappearing from the stock because of conversions, demolitions and rent increases, according to the coalition.

Although waiting lists for public housing are several years' long in many American cities and urban counties -- including Montgomery County, where more than 4,000 households are on the list and the wait is about 2 years -- HUD says plenty of housing is available.

One of HUD Secretary Samuel R. Pierce Jr.'s first acts when he took over the department five years ago was to commission two studies of the nation's housing stock, said Jayne Gallagher, head of the agency's public affairs office.

Both studies found there was no shortage of rental housing in the country, although there were some pockets of tight housing, she said. The administration has used these findings to justify ending the Section 8 housing construction programs, a major source of new units, and the decision to rely on the existing stock, in both public and private housing.

"We can house three households with vouchers for every one unit built with that Section 8 construction program," which was "extraordinarily expensive," Gallagher said. Using the vouchers, families find their own houses and HUD pays the difference between 30 percent of family income and the fair market rent for the area, established by the agency.

A potential source of money that could be used to house the poor is the $40 billion that homeowner tax deductions will cost the federal treasury in 1986, suggest some housing groups. More than 70 percent of these breaks go to taxpayers earning more than $30,000 annually, according to Congressional Budget Office and Treasury reports.

The latest threat to low-income housing is contained in the Gramm-Rudman-Hollings balanced budget legislation, according to low-income housing advocates. A provision of the bill, if approved as it now stands, could lead to cuts in funds already authorized by Congress and committed to specific housing programs, according to a letter sent this week to Senate and House members of a conference committee who are trying to resolve differences over the legislation. The letter was signed by the National Low-Income Housing Coalition and 11 other associations representing state and local housing officials and agencies.

Up to $45 billion in HUD funds may be in jeopardy, with the controversial provision posing "a fatal threat to over one million families receiving housing assistance and to over 100,000 units in the construction pipeline," the letter said.

The budget-balancing legislation would require automatic cuts from funds committed to programs but for which contracts have not been signed, if Congress and the president do not meet established spending cut goals contained in the bill. Much housing money falls into this category, because it has been put aside for assistance programs authorized for future years or for later stages of construction or operation of projects already under way, according to the low-income housing advocates.

A HUD spokesman said the legislation is under study because "it is not clear what that bill says."

"Virtually every interest group in America is painting a worst-case scenario that their programs" would be gutted by the bill, said Michael Fernandez, press secretary for Sen. Ernest F. Hollings (D-S.C.), one of the legislation's sponsors. "Where there are multiyear contracts with money to be paid out on a year-by-year basis . . . there is a possibility that some of those unobligated funds would be touched. . . . Where there is a contract in force, it would probably not be touched."

The national policy of housing the poor is long-standing. But the Reagan budget proposals "would shove aside a 50-year-long national commitment to help every American family secure a decent home in a suitable environment," charged the report for the Senate subcommittee.

Congress first voted for housing assistance in the 1937 Housing Act, when the federal government came to the rescue of Americans devastated by the Depression. Until the 1950s, all of the aid was in the form of public housing projects owned and operated by the government. The growth spurt did not come until the early 1970s. HUD's housing assistance budgets grew from $13 billion in 1975 to a peak of $32 billion in 1981, the last budget of the Carter administration.

Some of the housing aid of the early 1970s was well-intentioned but ill-conceived, consisting mainly of two types of interest-reduction programs. Under the Section 235 plan, a low-income family could buy a home, paying as much interest as they could afford on the loan, but never less than 1 percent. HUD paid the difference between what the buyer could afford and the interest on the loan, making monthly payments to the lender.

The Section 236 program worked the same way, with the interest supplements going to developers and owners of apartment buildings for low-income tenants. The monthly HUD payments made up the difference between what owners could afford, according to a formula based on estimated rents they could collect, and the cost of the loan.

The cost of Sections 235 and 236 "got out of hand," with the homeowner program turning into a financial disaster, because of the high rate of mortgage defaults, said Brent Shipp, a budget analyst with the Congressional Budget Office.

"This led to the theory that homeownership is not a good way to house poor people. An experiment in homeownership probably will not be tried again on a large basis," Shipp said.

An exception is rural housing, where many of the poor families eligible for aid are helped to buy homes. Advocates of the ownership program argue that there are few rental units available in rural areas, making rent supplements impractical.

Homeownership for the poor is an often-spoken-of goal of the Reagan administration, but its ownership programs serve only a small percentage of the low-income households in subsidized housing, according to administration critics.

The large majority of families receiving federal assistance are simply too poor to afford homeownership unless mortgages are heavily subsidized or their units are given to them, said Shipp.

After the unsuccessful homeownership programs, HUD turned to rent supplements. Known as Section 8, the program provided for construction of new apartments as well as for rent supplements to families in existing housing and became the mainstay of federal housing assistance.

Section 8, along with other aid programs for the poor, elderly and the handicapped, have taken big hits from the current administration.

Critics say HUD has mounted a three-pronged attack on housing aid: a fight to end all federal support for low-income construction, cuts in benefits and the number of families receiving them, and an attempt to cut the number of home loans and guarantees provided by the government.

Although HUD has not asked for any new construction funds, it has gone forward with 100,000 units that were "in the pipeline" from earlier administrations, said Thomas Sherman, director of the agency's office of public housing. About 80,000 of those units have been built or are under way, he said. Money for 5,000 new public housing units was approved Congress in 1985, although HUD did not want it.

"We emphasize putting to use existing units of public housing rather than letting them go to waste," said Warren T. Lindquist, HUD assistant secretary for public and Indian housing. Of the estimated 70,000 public housing units now vacant, about 25,000 are being rehabilitated and the rest are uninhabitable, some too badly deteriorated to make renovation practical, Lindquist said.

Since 1968, about $8.3 billion has been approved for modernization of public housing, with just over half the amount authorized during this administration, according to Lindquist. Results of a study commissioned two years ago are expected by the fall of 1986, and will include an estimate of how much more money is needed to finish the modernization, he said.

Housing vouchers, with which the poor are supposed to find their own homes in existing private housing, will provide less help to low-income families than past programs because they will be adjusted for inflation only twice in five years, and tenants also could be forced to pay more than 30 percent of their income, the ceiling on the amount of rent they had to pay with the Section 8 certificates, critics said.

Voucher users spend up to 30 percent of their income for housing, with the vouchers covering the difference between that amount and the fair market rent established by HUD. Unlike Section 8, families could find more expensive homes and pay the extra themselves. The voucher payment is always the difference between 30 percent of family income and HUD's fair market rent, so that a tenant who finds housing for less than the HUD payment standard can keep the savings.