Prospective homeowners who depend on loans with low down payments to finance their dream houses will find it tougher and more expensive to get such mortgages in the coming decade.

Tighter lending regulations and mounting troubles with government-insured mortgage funds are being blamed for the problem, which some housing industry analysts say will prevent thousands of potential buyers from owning a home.

Traditionally, lenders ask home buyers to make down payments of 20 percent or more of the purchase price of a home to obtain a mortgage for the balance. Lenders have found that buyers who put up that much equity are the least risky of borrowers.

Buyers who want to obtain mortgages with 5 percent or 10 percent down payments have been able to do so, however, either by qualifying for a government-insured Federal Housing Administration loan or by buying private mortgage insurance (PMI), which guarantees lenders against loss if there is a default.

But private lenders and the FHA are under fire these days.

"There is a somewhat uncertain future for low down-payment loans," said Sam Lyons, senior vice president of mortgage banking for Great Western Bank in Beverly Hills, Calif. "Yellow caution lights are already flashing, and we're getting mixed green and red signals from regulators."

Because the federal savings and loan bailout legislation placed higher capital requirements on private lenders, it has become more difficult and more costly to make low down-payment loans, Lyons told members of the National Association of Real Estate Editors recently in Raleigh, N.C.

"Lenders are studying cash equity and asking for more money from the borrowers up front," Lyons said. "Some lenders are going to stop making any loans other than those with at least an 80 percent loan-to-value ratio {a 20 percent down payment}."

Private mortgage insurers agree that it will cost buyers more for low down-payment loans, but they insist the loans still will be available.

"There are some real problems for all of us that can, we believe, be successfully dealt with," said William Simpson, president of Republic Mortgage Insurance Co., a private insurer based in Winston-Salem, N.C. "We think the future of low down-payment lending is still solid."

According to the Mortgage Insurance Cos. of America, a trade association of private mortgage insurers based in the District, insured mortgages made up 28 percent of the $357 million in home loans last year. Many of those buyers are first-time buyers making low down-payments, the trade group said.

The peak year for insured loans, both government backed and conventional, was 1985 when 36.5 percent of the $241.5 million in mortgages was insured.

Low down-payment loans are risky. Borrowers who only put 5 percent of the purchase price down on a home default at twice the rate of borrowers who make a 10 percent down payment, Lyons said, adding to lenders' reluctance to finance such purchases. On the other hand, Lyons said, Great Western's figures show that buyers who put 10 percent down have a delinquency rate only one half again as high as those who put down 20 percent or more.

"The 95 {percent loan-to-value mortgages, or 5 percent down} are still more risky" than 90 percent loans, said Charles Reid, president of United Guaranty Corp., a Greensboro, N.C.-based private mortgage insurer. "But there are a lot of places that exhibit characteristics where the 95s work a lot better.

"They are places where you don't get a red-hot local economy, there is not a lot of in-out migration, a moderate growth rate, rather steady housing appreciation -- plain vanilla regions that don't attract builders who overbuild and where people put down roots and take care of personal finances," he said.

Lenders were willing to make low down-payment loans more readily in the 1980s because they saw housing prices appreciating rapidly in many areas, meaning houses that might end up in foreclosure likely could be sold for far more than the mortgage amount. But appreciation has cooled in most markets in the country, and prices actually have fallen in some previously overheated areas of the Northeast, causing lenders to take a more realistic view.

The annual PMI payment on a $100,000 loan varies with the down payment. On average, Lyons said, with 10 percent down, a buyer will pay an initial insurance fee of $650 plus a $350 annual premium, for a total first-year cost of $1,000. With 5 percent down, the initial fee will be $1,000, and the annual premium $500, a first-year cost of $1,600. The premiums drop after 10 years.

Great Western, which does not issue loans with less than 10 percent down, does not require PMI for its 90 percent loan-to-value loans. Instead, the financial institution self-insures the mortgages by charging customers a higher interest rate, about one-quarter of a point, adding about $20 a month to the payment on a 30-year, $100,000 mortgage.

United Guaranty predicts it will have to pay mortgage insurance claims for bad loans on about 19 percent of the 5 percent down mortgages it issued in 1982 and 1983. About 11 percent of the loans issued with a 10 percent down-payment in those two years are expected to be foreclosed.

But the company projects that for the years 1990 through 2000 about 10.5 percent of the 5 percent down loans will go bad, while the default rate on 10 percent down loans will drop to about 6 percent.

Representatives of private mortgage insurance companies declined to predict how much their rates will rise. But they do acknowledge that either fees will go up or underwriting will get more stringent to reduce the risks of low down-payment lending.

Auditors examining the FHA program for Congress said that it, too, must tighten underwriting standards or face insolvency. The FHA insurance program lost $4.2 billion to bad loans in 1988, and its net worth has decreased to $2.6 billion from $8 billion 10 years ago.

As part of a comprehensive housing funding bill, the Senate has passed changes that would force borrowers to put more money down on the homes they buy or be asked to pay an added insurance premium over the base of 3.8 percent for loans with less than 5 percent down. And buyers would be forced to pay at least two-thirds of their closing costs up front rather than financing them over 30 years along with the purchase price .

Analysts for trade groups said a family using an FHA loan to buy a typical $75,000 house could expect to pay additional up-front costs of at least $1,100 if the new rules become law.