The savings and loan industry is winding up the year faced with some of the most serious problems in its history but buoyed by the prediction that its profits could be the highest on record, according to industry leaders and experts attending the annual convention of the U.S. League of Savings Institutions here this week.

This year's profits for the industry as a whole could reach $5 billion, substantially above the previous record of $3.9 billion set in 1978, the U.S. League predicted. However, 543 S&Ls still were operating at a loss at mid-year. New, tighter regulations on investments and growth imposed by the Federal Home Loan Bank Board, which regulates savings and loans, have helped curb the excesses of some thrift institutions this year, several experts said.

The prospects for next year are for a healthy housing market, with interest rates remaining around 11.5 to 12.5 percent for fixed-rate mortgages and housing starts staying in the 1985 range of 1.7 million, several said. They differed on the prospects for recession, with most saying none is in sight. But James W. Christian, the U.S. League's chief economist, predicted the economy will "muddle through its fourth year of expansion, but at a real rate of growth so anemic as to leave the onset of recession an ever-present danger."

The collapse of Equity Programs Investment Corp. (EPIC) this year left more than a score of thrift institutions in serious trouble and was a dramatic example of the S&L industry's real estate problems. The 60-day delinquency rate on construction loans held by many of the nation's approximately 3,500 savings and loans is averaging 8 percent, dramatically higher than the delinquency levels of 2 percent and 1.4 percent for fixed-rate mortgage loans and adjustable-rate mortgages, said Dennis Jacobe, senior vice president and director of research for the U.S. League.

Many of the delinquent loans are commercial and multifamily construction projects concentrated in the Sunbelt states, where the most overbuilt markets exist, he said. Commercial vacancies are high throughout the nation, with double-digit vacancy rates in most metropolitan areas.

The $7 billion worth of real estate that S&Ls have been forced to repossess in recent months, more than twice the average of past years, has been a major factor in the failure of many, he said. Regulators tightened up on real estate investments by thrifts after many got into trouble, he added.

The high delinquency and foreclosure rates on residential mortgages that have added dramatically to the problems of S&Ls were cited by convention speakers who called for more careful scrutiny of loans.

"Underwriting and appraisal standards must be significantly improved," said Donald Hovde, a member of the Federal Home Loan Bank Board.

A new "quality control system" for rating loans and assigning rankings of low, medium or high risk to them has been instituted by the Federal Home Loan Mortgage Corp., known as Freddie Mac, according to Bill Thomas, executive vice president of the corporation. Freddie Mac will use the system in making its decisions for purchasing loans from mortgage lenders. The corporation packages the loans into securities that it sells on the secondary market.

Factors considered in ranking loans are the lender, the ratio of the mortgage amount to the appraised value of the property, the state and region where the mortgaged property is located, and the age of the loan, its purpose and property type, Thomas said.

"Our research correlates with data collected by the mortgage insurance industry that LTVs of 90 to 95 percent (low down-payment loans) have a higher rate of default," he said.

The Federal National Mortgage Association, or Fannie Mae, which also buys mortgages and sells securities based on them in the secondary market, earlier this year tightened the underwriting standards for the loans it buys. Among other changes, the organization no longer buys loans for which the down payment was less than 10 percent of the sales price.

Fannie Mae's foreclosure losses will exceed $100 million this year, and mortgage insurers will lose about $700 million, according to David O. Maxwell, the organization's chairman.

These numbers do not include potential losses caused by the collapse of EPIC, added Maxwell, who is head of an investors' committee seeking a solution to the EPIC crisis.

"EPIC is the ultimate example of the way housing inflation can cover up the lack of genuine value," Maxwell said. "The whole scheme was based on rising prices. When prices fell, so did EPIC."

With 656 savings and loans still on the list of institutions regulators consider to have problems, the shaky condition of the Federal Savings and Loan Insurance Corp. (FSLIC) continued to be the industry's major concern. FSLIC insures individual deposits in federally chartered savings and loans for up to $100,000.

The incoming chairman of the U.S. League, Gerald J. Levy, announced at the convention that he is forming a task force to "develop solutions" to FSLIC's problems, saying the federal deposit insurance corporation "is the foundation of consumer confidence in our business." Levy is chairman and president of Guaranty Savings and Loan Association in Milwaukee.

The insurance fund has $3.2 million in unobligated reserves, which is the amount available to cover losses, according to recent testimony at congressional hearings. As a result, FSLIC has allowed S&Ls that are technically broke to continue operating because it cannot afford to take them over.

Formation of a new organization that will take over and dispose of $3.1 billion in bad loans and property FSLIC has acquired from failed savings and loans was announced at the Dallas convention by by bank board Chairman Edwin J. Gray. Many decisions on how the new S&L, called the Federal Asset Disposition Association (FADA), will work have not been made, but it probably will borrow money, using the foreclosed property and other assets as collateral, funnel $2.6 billion in cash to FSLIC and retain $500 million for operating expenses.

Although the new association will turn FSLIC's bad assets into cash, it will not provide the infusion of capital that many regulators and financial experts say is the only way to solve the insurance fund's problems. Gray emphasized that formation of FADA is only one of several measures that will be needed to strengthen FSLIC.

The U.S. League helped scuttle a Gray proposal for a special assessment of 1 percent of deposits in FSLIC-insured thrifts. The charge would have penalized well-managed thrifts and destroyed dozens of already floundering institutions, critics said.

Most S&L operators are unhappy with an assessment of one-eighth of 1 percent per quarter that the bank board already has levied and that will contribute $1 billion to FSLIC this year. "It means we are all paying" for the mistakes of a few, Levy said. "I don't want to go back to regulation, but I want to spot the problems sooner," he added.

Another bailout proposal, the merger of FSLIC with the larger, stronger Federal Deposit Insurance Corp. (FDIC), which insures bank deposits, also has drawn opposition from most S&L officials.

L. William Seidman, the recently named FDIC chairman, said in a speech at the league convention that he is not in favor of the merger either. But he suggested that a merger or the special assessment might be preferable to the other possible source of money -- the federal treasury. Operating with money appropriated by Congress "would be a step in the wrong direction and would give the government a legitimate operating right with respect to the fund itself," Seidman said.

"As far as the FDIC is concerned, we shall be ready to be of whatever aid we can be under the circumstances," Seidman said. "If Congress decides we should take additional actions to be of assistance, we will be prepared to do so. We approach this with an attitude of wanting to help and not one of seeking domination and control."

Economist Walter W. Heller said merger of the two insurance corporations is "worthy of serious consideration." But in the end, Congress "will have to back them up by appropriating money," he said.