U.S. bankers say that they are unlikely to meet Mexico's demands for new loans next year to help solve the country's debt problems. A denial of loans would be a serious setback for long-term efforts to restore Mexico's financial stability.

The banks' skepticism about Mexico could cast a cloud over efforts to deal with other Latin American debtors, because Mexico's handling of its debt has served as the model for the rest of the region since the debt crisis emerged as a major concern in 1982.

"Mexico was the country that started the whole process of solving the debt problem. It was Mexico's success that led to the euphoria and all this silly nonsense about the debt crisis being over," said a senior American bank official involved in negotiations with Mexico. "Now Mexico is going to be the leader again, but in the opposite direction. The whole game plan is blown out of the water."

The two other major Latin American debtors, Argentina and Brazil, have taken differing approaches to the problem. Argentina, which now owes about $48 billion, has adopted an austerity program and has begun receiving modest new bank loans, its first few in three years.

Brazil, the developing world's largest debtor with outstanding loans totaling $100 billion, has a shaky new government that has been unable to make the budget cuts necessary to hold down soaring inflation.

The Mexican government announced last month that it will need between $2 billion and $3 billion in new foreign loans in 1986, but the banks are wary of lending more money to a country that already has postponed repaying the $96 billion that it borrowed in the past, the bankers and U.S. officials said.

The banks are expected to grant new loans only if Mexico is making significant progress toward cutting its budget deficit and loosening restrictions on trade and foreign investment, the sources said.

The banks also are likely to require that Mexico reach a new standby loan agreement with the International Monetary Fund before lending money of their own, according to the sources. Such IMF accords are viewed as "seals of approval" because they fix specific requirements for individual countries in solving their economic problems. Mexico's current IMF loan accord expires early next year, but the country has fallen short of its targets for curbing inflation and trimming its deficit.

"It will be a difficult negotiation," said a U.S. official who monitors the Mexican economy. "It may well hinge on whether there is an IMF standby agreement."

In the past, both the U.S. government and the banks have praised Mexico's dealing with its debt. Mexico was the first big debtor to accept a harsh austerity program. The banks agreed to accept only interest payments on loans as Mexico did not have enough money to pay the principal.

The plan called for Mexico to resume borrowing from international banks by early 1986. In theory, the banks would extend new loans voluntarily because of improvements in the country's economic performance.

Instead, the banks are concerned that Mexico has not made sufficient structural changes in its economy to enable repayment of new loans when they come due in the 1990s, bankers and U.S. officials said.

The banks are also worried about Latin America's debts in general, because of rising pressures in the region to ease the burden of interest payments and thus permit more rapid economic growth.

The official position of the international bank committee that deals with Mexico is that the country will obtain the loans. This view was expressed last month by Citibank's William Rhodes, who is cochairman of the committee and has played a prominent role in restructuring Latin America's debts.

But other bankers, who asked to remain anonymous, said Rhodes' statements were "wishful thinking." One senior bank executive said the Mexicans "made the quick fixes in their economy, but they didn't have the follow-up."

Mexico's position on next year's loan negotiations is that the country is doing its best to solve its domestic economic problems, and that it is the victim of factors outside its control -- such as falling world oil prices and slow growth in the industrialized world. Without the new loans, Mexico would have to accept even more severe austerity measures or risk running out of money to keep up interest payments.

Some of the needed money could come from the IMF, the World Bank or other international lending organizations, but the commercial banks will have to contribute at least $1 billion, and probably more, to satisfy the Mexicans, according to U.S. banking sources. One U.S. banker said the Mexicans deliberately had underestimated their needs for next year and that the country might seek between $4 billion and $5 billion.

One of the biggest concerns of both the banks and the U.S. government is whether the administration of Mexican President Miguel de la Madrid will persevere in carrying out its ambitious but politically unpopular economic program.

Since midterm legislative and municipal elections in July, the government has adopted new austerity measures, including a devaluation of the peso and budget cutbacks.

In a break with past policies, it also moved to open up its economy to international markets by dropping the requirement that businesses obtain government licenses before importing most goods.

The government's long-term goal is to transform the Mexican economy into one similar to those of Taiwan or South Korea. These Asian nations have built up their economies by relying on private investment and raising exports, whereas Mexico has a large public sector, and its companies concentrate on selling to the domestic market.

The concern of both the banks and the U.S. government is that the Mexican administration will back off from its program because of domestic pressure. Unions want more public spending, and business is worried about trade liberalization because it means competition from abroad.

De la Madrid is heading into the second half of his six-year term, when Mexican presidents historically have favored big spending policies to win favor before they leave power. Mexican presidents are barred by law from succeeding themselves, and they tend to lose a grip on their own governments as their ministers jockey for the right to be the next chief executive.

"All the pressures are to abandon discipline when you're going to be out of power in three years," a U.S. official said.

On the other hand, de la Madrid and his advisers consistently have emphasized the need to pursue sound policies or face worse difficulties in the future.

"He doesn't want to be remembered as Mexico's Herbert Hoover," said Guy Erb, a Washington-based consultant on Mexican trade and investment.