Low-income families pay more for housing nearly half the time when they use vouchers, the Reagan administration's favorite subsidy program, instead of the older rental assistance certificates, according to a new report released by the Department of Housing and Urban Development.

Families and individuals using vouchers paid 8 percent, or $32 a month, more in rent on average and a larger percentage of their income than those with the certificates. The vouchers also cost the government an average of $23 more per family each month in higher assistance payments, the report said.

All voucher recipients receive the same subsidy: the difference between 30 percent of their income and the "fair market rent" established by HUD for their community. They shop for their own apartments, pay any amount of rent they choose, and if they take a unit costing less than the fair market rent, they can pocket the difference.

Certificate users, on the other hand, must rent units at or below the fair market rent, and the government pays only the difference between 30 percent of their incomes and the rent. Certificate recipients also must live in buildings approved by HUD.

A suspicious Congress has blocked Reagan administration attempts since 1982 to replace more and more of the rent assistance payments, known as Section 8 certificates, which last for 15 years, with the controversial five-year vouchers. Section 8 refers to the portion of the federal housing law that finances housing assistance for the poor. The report, covering the first year of a four-year study comparing the two subsidy programs, appears to confirm critics' fears that vouchers would eliminate safeguards against high rents contained in the certificate program and would fail to produce benefits predicted by the administration.

The report, by Abt Associates Inc. of Cambridge, Mass., covers 3,504 users of vouchers and certificates issued by 18 of the 20 public housing authorities being surveyed. City, county and state housing authorities administer federal assistance programs. The Houston and Los Angeles agencies are taking part in the study but started their programs later than other agencies and did not have data in time for the interim report.

The Montgomery County Housing Opportunities Commission, one of the study participants, has found that 46.5 percent of its 132 voucher users are paying more than 30 percent of their incomes for rent and the same number are paying less, according to Mary Jones, who administers the program for the commission. The remaining 7 percent paid exactly 30 percent of their incomes.

She said the sizes of the payments were reported to Abt Associates but were not retained in the commission's computer. Vouchers are costing the commission $27 more per family monthly than the certificates, Jones said.

Adjustments to the fair market rent, which determines the amount of subsidy renters will get, are being made only twice during the five-year period of a voucher, in contrast to yearly adjustments required for certificates. This difference could mean higher costs for low-income families who will have to absorb the difference if rents rise between adjustment periods, Jones said.

The 18 housing authorities covered by the report said 48 percent of the voucher users paid more than 31 percent of their incomes for rent, and one-fourth of these users paid more than 40 percent. Forty-one percent of the voucher users paid less than 30 percent of their incomes.

Nearly all the certificate holders -- 95 percent -- paid 30 percent or less of their incomes. Some paid slightly more because of local regulations governing welfare and housing assistance payments.

Half of the voucher users rented units costing more than the fair market rent for their community, while another 30 percent took homes costing between 90 percent and 100 percent of the fair market figure. The remaining 20 percent who found apartments for less than the fair market rent were thus able to keep part of the rent subsidy money for other expenses.

Whites, elderly people and families already living in good housing had more success than other groups in using vouchers, probably because they were able to stay where they already lived without having to find a new unit, according to the report.

Overall, both housing programs were equally successful in one respect, the report said. A total of 60 percent of voucher users and an equal number of certificate recipients were able to find housing, the report said. Critics have predicted that voucher users in areas with low vacancy rates would be unable to find housing, but so far only New York and Boston have reported success rates of less than 50 percent.

Voucher proponents argue that users have wider choices, encouraging them to shop for lower rents and homes that fit their needs, "removing what might be regarded as arbitrary and paternalistic restrictions on the housing that is 'suitable' for the poor," according to the report.

The "element of choice" is the key difference between certificates and vouchers, and the report's findings "indicate there is choice being exercised {and} that some people are choosing to rent up," said June Q. Koch, HUD assistant secretary for policy development and research. In most cases, "if a family is paying more, they are getting more" or better housing for their money, she added.

A major administration goal has been to replace low-income housing construction with certificates or vouchers, or a combination of the two, because "twice as many people can be served as under construction programs," according to Koch. HUD officials said there is no shortage of housing, except in a few areas of the country, and that vouchers will enable families to find units in the existing stock of most communities.

But Wayne Sherwood, of the Council of Large Public Housing Authorities, said vouchers are not producing the benefits predicted, such as providing more units for low-income families while lowering the cost to taxpayers in subsidy payments. "The idea that somehow vouchers are saving the taxpayers' money is silly," he said.

Even when tenants stayed in the same units but switched from certificates to vouchers, their rents increased in nearly half the cases, he said. The report showed that 16 percent of these tenants were dealt rent increases of more than $100 a month and another 16 percent had rent hikes of between $50 and $100 monthly.

During hearings held by the Senate Appropriations Committee's housing subcommittee earlier this year, Sen. William Proxmire (D-Wis.) said early experiences with vouchers indicated that only landlords were "better off."

Proxmire's primary concerns are the "budgetary impact" of voucher use and that "the program can leave consumers in the lurch," an aide said this week. The real fear of opponents is that the vouchers are "part of a hidden agenda" and the first step toward ending all low-income rental assistance, the aide said.