The Reagan administration is pressing ahead with its plan to impose user fees on a variety of federal housing programs, prompting a growing chorus of protest from industry and consumer groups.

The administration's budget, released this week, proposes to charge veterans who seek Veterans Administration mortgages a fee equal to 5 percent of the loan, and to raise the fees on Federal Housing Administration loans to 5 points.

Currently, veterans are charged 1 point and FHA borrowers 3.8 points. A point is equal to 1 percent of the loan amount.

These fees, if imposed, would mean that a veteran borrowing $60,000 would have to pay $3,000 to the VA, plus whatever lender fees and "discount points" -- charges imposed to increase the lender's yield from the loan -- the lender may require.

Thus, according to Robert J. Spiller, president of the Mortgage Bankers Association of America, "we are apt to see a veteran having to come up with 8 points on a VA loan . . . , and on that $60,000 mortgage, we are talking of total fees that will be just under $5,000."

Currently, veteran borrowers are not permitted to pay more than 2 points -- the current 1-point VA fee and a 1-point lender fee. If the new fee is adopted, the permissible total would rise to 6 points. VA officials noted this week that the fees can be financed, rather than paid in cash up front, but one noted that in the case of a 100-percent loan, the addition of the point would raise the loan amount to more than the value of the property, a situation neither lenders nor secondary market investors are likely to be happy with.

In addition, the Reagan budget proposes to raise the fee paid by lenders to the Government National Mortgage Association, which pools FHA and VA loans so that securities can be sold based on them, raising new capital for new loans. This boost is small -- to 0.15 percent from the current 0.06 percent -- but it would add $356.21 to the cost of the average VA loan because lenders pass these fees on to borrowers.

Finally, the budget would impose fees on organizations, such as the Federal National Mortgage Association and the Federal Home Loan Mortgage Corp., which have some connection with the federal government and which raise money in the credit markets.

The administration argues that there is implicit federal backing for these groups' debt, and that it is appropriate for the federal government to charge a fee for what amounts to the use of its name.

Initially, the fees would be equal to 0.05 percent on all debt and mortgage-backed securities issued. The fee on debt would rise in the fall of 1986 to 0.083 percent, while that on mortgage-backed securities would remain unchanged.

Since the fees would apply not just to new debt but to rollovers of old debt as well, they would fall particularly heavily on Fannie Mae, which has $85 billion outstanding that must be refinanced. The association's chairman, David O. Maxwell, estimated that "this retroactivity accounts for one-third of the fees Fannie Mae would have to pay from 1986 to 1990."

"This so-called 'fee' could cripple our corporation's ability to serve . . . the market, in a repeat of the 1982 recession period," Maxwell added in a speech last week. "In these high-interest rate periods, our refinancing costs skyrocket. Since the fee would be based on our volume of borrowings, the additional costs it imposes could prevent us from providing the liquidity lenders so badly need during bad times."

With the exception of the FHA and Ginnie Mae increases, which can be done administratively, the administration fee proposals require legislation. Lobbyists for affected groups are already combing Capitol Hill for support and finding, in the words of an official of the Mortgage Bankers, "for the most part, a very sympathetic audience."

Several officials noted that a proposal in last year's budget to change the VA's practices with respect to foreclosed properties went nowhere.

And Secretary of Housing and Urban Development Samuel R. Pierce Jr. said this week that "if VA does not get the legislation, we will not" raise the FHA fee.

Veterans groups are also turning up the heat. Richard Weidman of the Vietnam Veterans of America called the VA fee a "tax" and charged that the administration's true goal is to destroy the VA loan program.