Just hours before the 39 American hostages were to be released today from their 16-day ordeal in the hands of Shiite captors here, the fragile diplomatic arrangements that would have freed them hit an unexpected snag that has prolonged their captivity.

In an apparent effort to remove that snag, the State Department in Washington issued a one-sentence statement late Saturday night suggesting that the U.S. plans no punitive retaliation against Lebanon after the hostages are freed.

Shiite leader Nabih Berri and his associates here blamed the last-minute holdup on remarks made by President Reagan in Illinois yesterday calling the hijackers and kidnapers of Americans "thugs and murderers and barbarians" and suggesting that the United States might strike back once the hostages are released.

Berri said this necessitated new assurances that Lebanon will not suffer any U.S. retaliation. "We are still awaiting guarantees that no retaliatory strike will be undertaken after the hostages are released," Berri said. "Guarantees must be given to Syria. If these guarantees are provided tonight, they the hostages will be released tonight. If not, we are not in a hurry. Tomorrow, the day after, who knows?"

The State Department statement, released Saturday night, appeared to be a response to Berri's demand for new reassurance. The statement said:

"The United States reaffirms its longstanding support for the preservation of Lebanon, its government, its stability and its security and for the mitigation of the suffering of its people."

State Department officials refused to give any explanation of this new declaration, which could be read as a mitigation of threats made last Tuesday by the White House to blockade Lebanon's ports, close Beirut's airport or take other unspecified military action to retaliate for the hostage-taking.

In Washington and, to a lesser extent Damascus, a sharply different assessment of the cause of the breakdown was offered from Berri's. Earlier, Reagan administration officials said they believed that Berri is using Reagan's comments as a pretext to explain the delay while the real reason is that the Amal leader is still unable to force the original hijackers to turn over to his control four of the hostages being held separately.

Tonight, Amal security forces were reported to be looking all over west Beirut for the missing hostages amid signs of increasing tension between Amal and the more radical Hezbollah, or "Party of God," thought to be responsible for the hijacking.

Syrian sources, also questioning the timing of Reagan's remarks, said they may have provided the pro-Iranian Hezbollah group holding the four with a reason for refusing to hand them over to Berri's more moderate militia.

The last-minute snag appeared to catch everyone by surprise, including millions of Americans watching scenes on television that only hours earlier had shown the clearly relieved hostages at a farewell dinner at a Beirut hotel. Later, the hostages were shown in a sun-drenched schoolyard where they had gathered in expectation of boarding Red Cross buses waiting to take them to Damascus.

But by tonight, they were believed to have been dispersed once again into the Moslem slums of West Beirut, where they have been kept for two weeks.

In Washington, White House spokesman Larry Speakes told reporters shortly after 4 a.m. today that the buses carrying the American hostages "are now departing" for Damascus and President Reagan had scheduled a 9 a.m. statement confirming the hostages' release.

In Damascus, a top aide to President Hafez Assad announced a premature "happy end" to the drama, as a giant C141 Starlifter aircraft of the U.S. Air Force arrived there shortly after noon today to take the freed hostages on to Frankfurt, West Germany.

With Assad's personal prestige now at stake, U.S. officials expected the Syrian leader to intervene fairly quickly to overcome the snag and remained optimistic that the hostages would be released shortly.

In Jerusalem, speaking before the breakdown and when it appeared the release was certain, Prime Minister Shimon Peres made the clearest statement yet that Israel intended to free the Arab prisoners it has been holding.

Peres said freeing the hostages would remove "obstacles" to Israel's planned release of its prisoners. Release of the 735 prisoners, most of them Lebanese Shiites, is the main demand of the hijackers.

In Geneva, Vice President Bush delayed his departure for Frankfurt, where he was to greet the hostages.

According to Arab sources close to the negotiations, Reagan's remarks had provoked a genuine stumbling block. They said Reagan's tough language had provided a pretext for the original hijackers, linked to Hezbollah, to stymie negotiations that seemed to have gone far beyond their control.

Jaafar Jalabi, a spokesman for Berri, put it this way: "For days, the Reagan administration said nothing. No comment. But for Reagan himself at the most crucial moment to make the kind of speech he made in Chicago is incomprehensible to them," meaning the hijackers. The spokesman described the hijackers "as extremely nervous people."

But later another Amal spokesman acknowledged that there were "political problems" with Hezbollah that were preventing Amal from taking control of the four missing hostages.

The hostages who are in the hands of the original hijackers were identified after a televised roll call of the 35 gathered this morning in the schoolyard near Beirut airport under Berri's protection as Robert Brown, 42, of Stowe, Mass; Richard Herzberg, 33, of Norfolk; Jeffrey Ingalls, 24, of Virginia Beach, and Robert Trautmann, 37, of Laredo, Tex. Two are military personnel and two civilians.

At 9:15 tonight, after a day of arduous negotiations, Berri went to bed telling his associates that he had little expectation the crisis can be resolved until Sunday. A delegation of Syrian soldiers who were supposed to have accompanied the hostages in a caravan of Red Cross ambulances to the border were given the same gloomy assessment, according to sources close to Berri. Later seven Red Cross cars without passengers were seen driving back into the city from the suburbs where the hostages have been held.

The hostages were in a bright mood, expecting to be set free at almost any moment, when they gathered at the schoolhouse near the Burj al Barajinah refugee camp.

They had loaded their bags -- still bearing TWA letters -- into a blue pickup truck and chatted casually with their guards and then with reporters who found them at the scene.

"We're on the way home," said the Rev. James W. McLoughlin, 45, of Geneva, Ill. "We're going east but it will eventually bring us west to the United States."

Another hostage, Steve Willet, 36, of Choupic, La., remarked, "We're relieved that we're going to be in the hands of a government now. It's going to help our position a lot, we hope."

Stuart Dahl, 31, a sailor from Norfolk, said he was among the original group kept separately because "they thought we were marines. An unusual experience. They put us in a safe area so we wouldn't hurt them."

In contrast to the optimistic mood in the schoolyard, in Berri's office fears were growing that the entire deal for the release could crumble.

Berri spent much of the afternoon closeted with advisers waiting for a call from Syrian Vice President Abdul Halim Khaddam to reassure him that there would be no U.S. retaliation.

Berri's associates said that once such guarantees are given it will be up to leaders of Hezbollah and the hijackers to release the four separated Americans and allow the entire group of 39 hostages to proceed to Damascus.

The situation has grown increasingly complex as the hostage drama emerges now at the center of the growing confrontation between Berri's Amal militia and the more radical, fundamentalist Hezbollah.

According to one source close to Berri, as soon as he got wind of the hijackers' reluctance to follow through with the planned release, Berri sent his forces to the TWA plane to remove the crew. The same source said that there was nearly a fight as a result, but the crew subsequently was taken to join the main group.

As far as is known tonight, the 35 hostages filmed today remain under the protection of Berri's Amal militia. However, sources close to the negotiations emphasize that the four believed held by Hezbollah effectively give them leverage against both the United States and Berri and could be used to demand the return of the other 35 to the hands of the hijackers.

A statement issued by Hezbollah yesterday declared solidarity with the hijackers of the TWA aircraft and warned against the possibility of American retaliation.

It termed the hijackers' demands that Israel release all the Lebanese detainees in Atlit Prison as "just and legitimate" and suggested that Hezbollah may be capable of striking inside America itself in response to any U.S. retaliation.

As the crisis lingered this evening, a Hezbollah delegation was reported by sources involved in the negotiations to have traveled to Damascus. It was believed by these sources that Hezbollah representatives would speak there not only with Syrians but with Iranian officials.

According to one source familiar with the negotiations, Iran has given its support to the arrangement made to free the hostages, but its allies in Hezbollah are reluctant to go along.

These sources suggested that Hezbollah had not been happy with the course of the negotiations as handled by Amal leader Berri since he effectively took charge early in the crisis.

Israeli government sources said privately today that the key concession made by Amal, clearing the way for an "understanding" was acceptance of the idea that there did not have to be a simultaneous release of the American hostages and the prisoners held in Israel.