The Reagan administration would like to get almost entirely out of the business of subsidizing housing for the poor by 1984 or 1985, Housing and Urban Development Secretary Samuel Pierce said yesterday.

Pierce also acknowledged that a proposed system of housing vouchers could serve only a portion of the "truly needy" to help them meet basic housing costs. HUD and the President's Commission on Housing envision replacing the federal government's main low-income housing program, which subsidizes construction, with the voucher system.

At the same time, the HUD secretary endorsed continued federal tax subsidies, including a few new and costly ones, for homeowners.

"We hope by 1984 or '85, that we will have interest rates down enough that it will stimulate housing so that we won't have to use the voucher system. We hope that maybe we'll even get out of that," Pierce told a group of Washington Post reporters and editors.

Housing vouchers are direct assistance to individuals, similar to food stamps, that HUD wants to see replace its Section 8 program that subsidizes builders of low-income housing. The department also wants to restrict eligibility to the poorest of the poor -- and knows not all of those can be served because of budget restraints.

"All of the truly needy will not get vouchers. I'll say that in a minute," Pierce said yesterday. It would not be designed as an entitlement program in which eligibility is defined and everyone who falls under the definition would be guaranteed the assistance, he added.

Section 8 housing is costly and involves a long-term commitment of future budget monies to a builder, Pierce said, and he conceded that one of the advantages of replacing Section 8 with vouchers is that they could be ended quickly.

Private groups that represent users of low-income housing have said this is their main fear about the voucher system -- that it may be just the first step to the total elimination of federal housing assistance to the poor.

Asked how he would achieve a phase-out of vouchers, Pierce said income limits for eligibility could be reduced each year.

Later in the wide-ranging luncheon discussion, however, Pierce backed down some from total elimination of assistance, saying the idea of ending vouchers is just "a possibility" and that if a need still exists, the administration would want to "do something" about it, either continue vouchers or start some other program.

Pierce also announced a victory for proponents of controversial mortgage revenue bonds. He said he has convinced the Treasury Department to support the use of the tax-exempt bonds by state and local governments to bring mortage interest rates down a notch or two for a few of their citizens buying single-family homes. The fate of the bonds for multifamily housing is still unresolved, though he is pushing that at Treasury, too, the secretary said.

The issue has pitted local governments against federal budget-cutters over the past few years. The tax-exempt status of interest on the bonds subsidizes rates for the lucky few who receive the low-interest money. The Carter administration wanted to end their use entirely, citing their costs to the federal government as potentially astronomical.

Asked about arguments that revenue bonds are costly and inefficient, Pierce said: "Sure, this is a false stimulation for the market that we're in... We need this to get the building we need. Until interest rates come down, we think we have to have it to meet the housing needs of this country."

Low-income or elderly renters who are displaced by conversions of apartment buildings to condominiums cannot expect aid or comfort from HUD in this administration, however.

"I believe in the marketplace on this [displacement]. You could give me some very said pictures on this I'm sure, but on the whole that has to be governed by the market," the secretary said. "I don't think the government should get involved."