If you are a veteran, a moderate-income home buyer or you're thinking about refinancing your mortgage, keep your eye on the federal-budget brouhaha that's breaking out on Capitol Hill.

You may have a direct stake in it, to the tune of thousands of dollars in cash. Here's what's happening in brief:

The battle of the federal deficit -- which previously touched only low-income housing-subsidy programs -- spilled over into the ranks of middle-American home buyers last week. The Reagan administration proposed what critics called a "housing-tax" program through sharp new restrictions on the use of Federal Housing Administration (FHA), Veterans Administration (VA) and other federally related financing aids.

The VA changes are likely to have the most immediate impacts. An estimated 270,000 veterans nationwide were expected to seek low-down-payment mortgage loans during coming years. As of March 1, however, VA will:

*Refuse to accept applications for refinancing. That means, in effect, that veterans with loans in the 13 percent range and above who had planned to refinance into a 10 1/2-to-11 percent VA-backed mortgage will be out in the cold unless they act fast. Fully one-fifth of the agency's typical annual loan volume is in refinancings of servicemen's existing VA loans.

*Prohibit use of VA financing in connection with loans larger than $90,000. The current national limits range from $110,000 for loans with no down payments to $135,000 for greater down payments.

*Turn away veterans who have partially used up their VA entitlements and want additional mortgage money to buy a larger house.

In addition to the March 1 cutbacks, the fiscal 1987 federal budget proposal sent to Congress by President Reagan contained further restrictions on VA. The White House asked lawmakers to raise the guaranty insurance fee charged home buyers on VA loans to 3.8 percent, up from the present 1 percent.

On an average VA-financed house priced at $62,600, according to calculations made by the National Association of Home Builders, that would require a veteran to come up with $2,380 for the VA fee, compared with just $626 today. The White House also would seek another $125 from the buyer of a typical VA-financed house in the form of a new levy on behalf of the Government National Mortgage Association (Ginnie Mae), which pools VA loans for sale to Wall Street investors.

If paid in cash at closing, the White House's budget plans would more than double the typical veteran's out-of-pocket costs of buying an average house to nearly $4,700, up from about $2,200 today.

The proposed cutbacks on middle-income home buyers using low-down-payment FHA mortgages would have similar effects on buyers' wallets. The new federal budget calls for an increase in the FHA insurance premium to 5 percent, up from 3.8 percent, plus a requirement that it be paid in cash at closing.

On a $59,500 house -- about the national average for FHA financings -- that would mean $2,851 of out-of-pocket costs at closing. Under current FHA rules, the 3.8 percent mortgage-insurance premium ($2,210) could be "financed" -- that is, tucked into the mortgage-principal debt and paid off over the full duration of the loan. The out-of-pocket cost at closing would be zero.

The amount of cash that an average FHA buyer would need to bring to the settlement table would exceed $8,000 under the Reagan budget proposals, according to home-builder estimates. Under current FHA rules, by contrast, average buyers need less than half that much, about $3,990.

The net effect of the proposed FHA budget, in short, appears to tilt the program toward buyers with higher incomes and greater family financial resources. But that's not what the White House had in mind. In a move designed to pinch the FHA from both ends, the budget calls for imposition of family-income limits for the first time in history.

Families with annual incomes of $40,000 or more would be prohibited from obtaining FHA-insured mortgages. Currently, one-third of FHA's annual 450,000 borrowers have incomes beyond that threshold. They would be denied the consumer protections that traditionally have made FHA mortgages so attractive, including full assumability of low-interest-rate loans by new buyers, no prepayment penalties and favorable refinancing terms, among others.

The Reagan budget attacks other pillars of housing financing as well. It asks for new annual taxes on the borrowings of the Federal National Mortgage Association (Fannie Mae) and the Federal Home Loan Mortgage Corp. (Freddie Mac). The levies, according to Fannie Mae officials, would be passed directly to the hundreds of thousands of conventional (non-FHA-VA) home buyers whose loans are purchased by these mega-lenders each year. The net effect would be a one-half of one percent mortgage-interest-rate increase for new-home purchasers, once the taxes were fully imposed two years hence.

The outlook on Capitol Hill for this wide-ranging, bad-news budget package for home buyers? Stay tuned. An election-bound Congress clearly cannot accept all the Reagan cuts aimed at average purchasers. But neither can it politically afford to say no to every one of them.