PANAMA CITY, JULY 9 -- A month of turmoil has damaged Panama's image as a peaceful haven for international banking and has dimmed its prospects for keeping up payments on its foreign debt, according to bankers and economists here.

To date there is no sign of panic among the 120 foreign banks, with assets of nearly $39 billion, that make Panama the most important international banking center in Latin America. No bank is slashing operations or relocating.

But as the crisis moved into its fifth week with no end in sight, bankers sent alarm signals to their home offices and the Panamanian government.

"You're starting to hear bankers reevaluating a few things," said one foreign economist.

Banks in Panama handle off-shore accounts for American companies, Latin funds and bank-to-bank transactions. The banking center flourished over the past decade because of strict secrecy laws and negligible taxes on deposits and income. U.S. dollar bills are used as the local currency. An atmosphere of political calm compared to other Latin nations was also an important factor -- until June.

Now the floor-to-ceiling windows of most banks along 50th Street, the financial district's main boulevard, are boarded over to protect against pro-government rioters throwing rocks and firebombs.

The unrest erupted last month after defense forces Chief of Staff Col. Roberto Diaz Herrera was forced out by strongman Gen. Manuel Antonio Noriega. Diaz Herrera publicly accused Noriega, who in practice controls the government, of being involved in assassination and of rigging a 1984 election.

A new round of pro- and anti-Noriega demonstrations came after a June 26 U.S. Senate resolution calling on the general to step aside pending an independent investigation of Diaz Herrera's charges.

All banks were closed June 11 and 12 during a nationwide business strike, mainly because their employes didn't show up for work. The government threatened to cancel the visas of some Latin bankers based here if they did not reopen after those two days, the bankers said.

Bank staffers have been among the most conspicuous opposition activists, taking daily to the streets to honk horns and wave white handkerchiefs to demand that Noriega step down.

As a result, many banks were targeted by police and pro-government vandals. During protests in mid-June, police stormed the offices of the Panamanian-owned Banco del Istmo, clubbing several employes. One of the bank's main stockholders is President Eric Arturo Delvalle.

Last week pro-government squads broke the Bank of America's ground floor windows and tossed a Molotov cocktail into a second-floor office, although it didn't ignite. They also splattered paint across Citibank's facade. Police arrested four Chase Manhattan employes and beat several others.

One major international bank lowered Panama one grade on a political risk scale, meaning the bank will exercise much greater caution with its loans here. One bank executive reported a string of withdrawals by nervous foreign firms totaling $10 million to $20 million.

Panamanian depositors withdrew millions in cash from individual accounts in local banks. In June, fiscal authorities ordered several emergency shipments of dollar bills, worth $20 million, from commercial banks in the United States, a bank treasurer said.

Bankers noted that since Panama has no local paper currency and, therefore, no central bank, a liquidity shortfall in a small bank can easily snowball.

"The longer it goes on, the more worried we get," said one international bank manager. "If it continues for another week with violent confrontations, we're going to be in a difficult situation."

Adding to the jitters is Luis Alberto Arias' abrupt resignation with no explanation July 3 from his post as director of Banco Nacional, the government bank. The move by Arias, who held senior positions in the bank for a decade, fueled speculation he had been pressed to overdraw funds for the government.

Wednesday, the harried executive secretary of the government's National Banking Commission, Mario De Diego, sought to reassure businessmen by calling rumors the government planned to nationalize banks "preposterous."

In a parallel problem, bankers monitoring the $3.8 billion foreign debt said the crisis greatly complicated Panama's compliance this year with a World Bank austerity program in order to qualify for a $50 million bailout loan.

Panama agreed in 1985 with the World Bank to reform its social security system, which was close to running out of funds. In the past two years, Panama has met other World Bank and International Monetary Fund program requirements and has received loans totaling almost $200 million. It owes $420 million in 1987 on the debt.

But there is widespread resistance to the social security reform, which would make many Panamanians work longer for smaller pensions.

"Now it will be very, very tough for the government to pass an unpopular measure, especially one which is strongly opposed by the forces now supporting it," said one banker familiar with the debt talks. Without a World Bank program, he said, "The banks will be very upset. They will show a lot of resistance to making further adjustments."

Bankers said government economic officials are aware of the serious bind but promise little response on their own because they know that only Noriega is in a position to call the shots.