The critical moment in the intricate diplomatic ballet that produced freedom yesterday for 39 American hostages in Beirut occurred sometime late last Tuesday or early Wednesday, when Shiite Moslem leader Nabih Berri changed his mind.

Until that point, Berri seemed disinclined to accept any formula for ending the hijacking that did not include release of more than 700 Lebanese -- mostly Shiites -- held prisoner in Israel before the Americans in Beirut could go free. But on Wednesday morning in Beirut, according to U.S. and foreign sources familiar with details of the last 17 days of international maneuvering, Berri was suddenly much more flexible.

A White House official involved in the discussions credited the breakthrough to a variety of internal and international pressures on Berri, including the threats from President Reagan last Tuesday to close down Beirut airport and take other unspecified reprisals if diplomacy did not succeed within several days.

Another interpretation of Berri's change of heart gave more emphasis to the vital role of Syrian President Hafez Assad, who personally entered the hostage diplomacy only after returning from a trip to Moscow a week ago. U.S. officials believe that Berri was summoned secretly to Damascus by Assad last Tuesday night, before the White House had announced its threats to retaliate. In the Syrian capital, Berri was strongly encouraged to find a way out.

The diplomacy that finally won freedom for the U.S. hostages began in the first hours after the hijacking of the TWA jet on June 14, and involved U.S. contacts with traditional foes such as Iran and the Soviet Union as well as friends such as Israel, Algeria, Saudi Arabia, France and Switzerland.

In the end, the key player was Syria, which is seen sometimes as a friend but more often as a foe by the Reagan administration. Assad, who two years ago played a crucial role in forcing the United States out of Lebanon, used his considerable power there to help the United States free its citizens. During this episode, Assad discovered that he shared a common interest with the United States, and even with his hated enemy, Israel -- to prop up Amal leader Berri against more radical Moslem Shiites backed by Iran.

When the release of the Americans was unexpectedly jeopardized Saturday by the refusal of Hezbollah, a radical Shiite faction, to accept the deal worked out by Assad and Berri, Assad reportedly dispatched his chief of intelligence for Lebanon, Col. Ghazi Kanaan, to Hezbollah leaders for some unvarnished persuasion. At the same time Damascus suggested a public statement from Washington vaguely disavowing any American interest in destabilizing Lebanon that made acceptance of the arrangement more palatable to the Hezbollah radicals.

At the outset of the hijacking U.S. diplomacy centered on Algeria. On the first two days, the White House made fervent pleas to Algiers that TWA Flight 847 be allowed to land in Algeria and denied permission to take off again. The Algerians did allow the plane to land twice, but each time permitted it to depart again at the hijackers' demand.

The Reagan administration considered using American force to keep the plane in Algiers, but decided this would be politically impossible. "It would have destroyed our relations with Algeria," said a State Department official.

By Sunday, June 16, when the jet landed for the third and final time at Beirut airport, which is controlled by Berri's Amal militia, both the Amal and Hezbollah factions had armed men on board. An explicit deal was made between the two groups, according to U.S. officials, that Berri would take control of the situation and that hostages would not be harmed, but that the Americans would not be released without obtaining the Shiites and other Lebanese Arabs held by Israel.

Reagan's first meeting on the situation with his senior national security advisers took place at the White House that Sunday afternoon. Those present shared the gloomy realization that what began as a hijacking had become an even more difficult and longer-lasting hostage problem reminiscent of the 1979 seizure of Americans in Iran.

Perhaps the key decision at this initial meeting was that the United States would not make or arrange a deal to swap Israel's prisoners for the American hostages. Secretary of State George P. Shultz, who was adamant on this point, argued that giving in to terrorist demands would leave other Americans vulnerable to such pressures in the future. "We will not make deals with terrorists and will not encourage others to do so," Shultz declared the following day.

The administration was faced with a power struggle on the ground between Berri's Amal and the more radical Hezbollah, the intensely emotional demands of both groups for release of their fellow Shiites in Israel -- and all of this in a chaotic Lebanon, a cockpit of bloody conflict involving opposing sects of Lebanese as well as Palestinians, Iranians, Syrians, Israelis and others.

Berri and the original hijackers had to be convinced that their demand for prior release of the Lebanese prisoners in Israel could not be met. At the same time, the administration did not discourage the possibility that when the Americans were freed, the Lebanese could be released by Israel in due course. But the link, to protect the U.S. position of no concessions to terrorists, could be only "inexplicit and indirect," as one official put it.

U.S. officials and foreign diplomats who dealt with them said yesterday that throughout the 17-day ordeal, Washington insisted privately as well as publicly that there could be no "linkage" between release of the American hostages and the Shiite prisoners.

"Assad came to understand the U.S. position on this point," said a diplomat familiar with the Syrian position. Equally important, Assad also "understood," the diplomat said, that the United States would vigorously press for early release of the Shiite prisoners by Israel, if such pressure were needed after the American hostages were freed.

The discussions with Israel, the full story of which is not yet known, were delicate and at times difficult. Though the U.S. administration was determined to avoid linking the freeing of the American hostages to release of Israel's prisoners, officials in Jerusalem in effect had done so in the early days of the hijacking, when they made it known that they would release their prisoners if an official request came from the Reagan administration.

Washington wanted Israel to be ready in time to release the Shiite prisoners -- as Israeli authorities had promised they would. The Reagan administration had formally criticized Israel for holding them in early April. And despite its official stand against "linkage," the administration permitted and in some cases encouraged the buildup of public pressure for Israel to make this release, once the U.S. hostages were freed.

The administration sought to bring pressure on Syria and the Lebanese Shiite factions for unconditional release of the Americans through diplomatic appeals to the Soviet Union, Iran, Saudi Arabia and a variety of European countries with Mideast ties.

The response of the Soviets, Syria's superpower ally, was not known. (Publicly, the Soviets said they would like to help solve the problem.) Iran, which is believed by some to have encouraged the hijacking and by many to encourage such acts in general, gave an unsympathetic response to a U.S. diplomatic message through the Swiss, but is believed to have acquiesced in the settlement. The Saudis were instrumental in passing messages, and sent unofficial emissary Rafiq Hariri to Beirut and Damascus.

In the end, Berri's weakness may have been his best asset. Both Assad and the Israelis looked to him as an important ally in Lebanon, and neither seemed prepared to let him fall on his face. For his own part, Berri must have seen Assad as his ticket out of a mess that he had not created nor anticipated when he agreed to take responsibility for the passengers on Flight 847.

Assad offered Berri a way out: he would give assurances that the Israelis would indeed release their Lebanese prisoners, and he could provide the muscle to enforce a compromise deal on Berri's Hezbollah rivals. It became an offer Berri could not refuse.