The backlog of Federal Housing Administration mortgage insurance applications continued to mount this week as Congress failed to reach an agreement on legislation to revive the popular program, which now has been out of business two weeks.

The FHA's operating authority expired for the sixth time this fiscal year at midnight June 5, and the dollar limit on the amount of mortgages it can insure was reached Wednesday. Several bills are stalled by disputes on what is to be included in the final legislation, and congressional and industry sources predicted no reauthorization measure will pass before next week at the earliest.

Because authorizing legislation for the mortgage insurance program is certain to be approved by Congress and signed by the president, less popular legislation frequently is attached to it. The traditional tussle has been more intense this year, and longer and more frequent lapses in the agency's operations have resulted.

When the FHA hit its credit ceiling of $74.2 billion this week, field offices were told to stop assigning case numbers, the first step in handling an application, according to FHA Administrator Silvio DeBartolomeis. The offices had continued to process requests for insurance after FHA operating authority expired, although applications received after June 5 could not be given final approval.

About 60 percent of FHA home buyers are purchasing their first home; the average income of all purchasers of FHA-insured houses is $37,000. The program, created in 1934 to help moderate-income Americans buy homes, has issued nearly $342 billion worth of mortgage insurance. There is no limit on borrowers' incomes, but the loan amount that can be insured is capped at $67,500 except in high-cost areas where the maximum can range up to $90,000.

As the delay in loan approvals lengthens, members of Congress, HUD officials and lenders are reporting an avalanche of calls from upset home buyers and sellers whose transactions are being held up because loans cannot be approved.

DeBartolomeis estimates that between 5,000 and 6,000 people each day are unable to close FHA-insured loans that were in the pipeline before the agency's operating authority ran out.

Many additional buyers and sellers, who are not waiting for FHA loan approval, also are being affected. Fairfax resident Maggie Nenner said she is a "third-party-removed" victim of the shutdown. She and her husband sold their house to a family who, in turn, sold their home to a woman who is awaiting approval of her FHA-insured loan.

A home-sales explosion fueled by declining interest rates has inundated FHA offices with up to 10,000 applications daily in recent months, meaning that as many as 100,000 people already may have been denied loans, according to the Mortgage Bankers Association of America.

"We are in the midst of a housing boom and the government has gone out of the housing business," said Warren Lasko, executive vice president of the MBA. He said about 3,000 prospective buyers a day are seeing their lenders' 60-day interest rate guarantees expire because loans cannot be completed. For many, the rates will go up as much as 1 1/2 percent because of the recent rise in market interest rates, he said.

The Washington area FHA staff, the busiest in the country, has 3,000 applications they have not yet started processing, according to Margart White, head of the office. Another 20,000 insurance applications were waiting for appraisals to be completed when the order came to shut down operations. White said the heavy rate of buying and mortgage refinancing has pushed the two months normally needed to process a loan up to four months, and the applications will be backed up even more when the agency can go back to work.

Relief will not come until action is taken in Congress, where the "FHA is being held hostage to the squabble over the housing bill and other issues," Lasko said.

Lawmakers who favor more spending on housing programs than the bare-bones budget of $5.5 billion the Reagan administration proposed have tried to use legislation rescuing the FHA and the Urban Development Action Grant program as a lever to get a full housing bill through the Republican-controlled Senate.

As a result, only short-term extensions of FHA have won approval since the start of the fiscal year last October, and efforts to attach UDAG changes to any FHA extension bills have been resisted. In addition to keeping UDAG, which the administration wants to kill, several senators from the South and West, including Strom Thurmond (R-S.C.) and Charles E. Grassley (R-Iowa), want changes in criteria for awarding the grants that will channel more money to their states.

A bill extending the FHA insuring authority through June 27 was passed by the House last week. The Senate passed the measure this week, after amendments extended the authority to July 25 and put a freeze on making UDAG awards. The House is expected to balk at the UDAG provision when the bill goes to a House-Senate conference.

Rep. Chalmers Wylie (R-Ohio) introduced legislation this week extending FHA operating authority through September 1987, raising the credit limit to $132 billion for the fiscal year ending Sept. 30 and setting a ceiling of $100 billion for fiscal 1987. It does not mention UDAG, and is expected to draw opposition in the Senate.

House and Senate conferees this week agreed on a supplemental appropriations bill that included an FHA extension until July 25 and an increase in the credit ceiling to $132 billion. The appropriations measure contains provisions the White House opposes, however, and President Reagan has said he may veto it.

The House passed a $16.3 billion housing authorization bill last week, with provisions renewing FHA insuring authority through September 1987, reviving UDAG and changing criteria so more funds can flow to southern and western states. The Senate housing bill, which also would extend FHA authority until September 1987, is not expected to be acted on until next week. The two measures would have to go to a House-Senate conference before going to the White House for the president's signature.