In a confidential message to Washington last Tuesday, Syrian President Hafez Assad proposed the artful diplomatic formulation that ultimately resolved the conflict between the hijackers' demands for release of Israeli prisoners and the Reagan administration's refusal to make concessions to terrorists, U.S. and diplomatic sources said.

The Assad message, which followed a flurry of exchanges between Washington and Damascus in the previous day or two, carefully avoided asking for a formal commitment that Israel would release the 735 Lebanese Shiite and other Arab prisoners it held in return for freeing the 39 American hostages held in Beirut.

Instead, these sources said, Assad offered to take the problem of "linkage" on his own shoulders. He stated his willingness both to accept custody of the Americans and then release them, and to give the hijackers a guarantee of his own that Israel would release its Lebanese prisoners.

"He informed us what he would do and simply asked, 'Is this okay?' " said an administration official.

Within a few hours -- after what one source said was a telephone conversation between Secretary of State George P. Shultz and Israeli Prime Minister Shimon Peres -- the United States informed Assad in guarded language that there was no objection in Washington to the course he proposed to take.

As a result of this tacit arrangement, Assad could go forward to accept and then release the American hostages last Sunday with a well-founded understanding that Israel would then release its Lebanese prisoners. At the same time, the United States and Israel decided they could credibly insist that "no deals, no concessions" were made to the hijackers or Lebanese Shiite leader Nabih Berri, who took over responsibility for the Americans in the early days after the hijacking.

The timing of the message from Assad casts a new light on the White House announcement late Tuesday that Reagan was prepared to take retaliatory steps including military action if diplomacy did not succeed in freeing the hostages within several days.

One explanation given by official sources for the decision to threaten retaliation on the same day that Assad was offering a diplomatic solution is domestic politics. The threatening White House statement was intended as a response to rising public and political demands that Reagan "do something" in response to the hijacking and hostage-taking, according to some sources.

Another explanation is that despite Assad's adroitly worded offer, administration officials did not know on Tuesday whether he would be able or willing to follow through. In fact, according to one informed source, nothing authoritative was heard from Assad about his proposal from Tuesday until last Friday evening, when word came that the hostages would be sent to Syria and released within hours of their arrival.

Still another explanation of Reagan's threats of reprisals last Tuesday, most explicitly the threat to close Beirut airport and cut off Lebanese ports through a naval blockade, is that they were intended to increase the pressure on Lebanese Shiite leader Berri to accede to release of the Americans. This explanation was offered by a senior White House official in a meeting with reporters as the hostages were being released Sunday afternoon.

In his speech to the nation Sunday night, Reagan said that "Syria has had a central responsibility" for the release of the Americans. About the same time, Reagan sent a message of thanks to Assad as the 39 Americans flew out of Syrian airspace aboard a U.S. Air Force plane on the first leg of their journey home.

Aides to Assad expressed displeasure Monday with what they described as a lack of U.S. gratitude for Syria's role in freeing the Americans. The White House disclosed yesterday that Reagan spoke by telephone with Assad for about 15 minutes Monday thanking him and also asking that he now use his apparently considerable influence in Lebanon to win release of seven other Americans held hostage there. It was not clear whether the call was made before or after Syrian officials complained of a lack of American gratitude.

Reagan previously appealed in confidential messages for Assad's help in freeing the seven kidnaping victims, who are believed held by several different Moslem extremist groups in more than one location. The Syrian president reportedly committed himself to do everything he can.

U.S. sources said they believe Syrian forces have been able to identify sites where the kidnaping victims are or have been held. But they said the Syrians concluded that to free them without their captors' consent would require military action that could result in injury or death for the Americans.

Some administration officials said they believe that Iranian authorities, who have a close relationship with the most militant Shiite groups, used their influence late last week to persuade the extremist Hezbollah, or Party of God, to submit to Assad's authority and release four TWA hijacking hostages the group was holding.

"We think the Iranians did help," said a U.S. source. This belief has given rise to hope that the same thing might be arranged in the case of the kidnaping victims. But this is only a hope.

The captors of the seven Americans are believed to be associated with the radical Hezbollah group, which held some of the TWA hostages and which created an 11th-hour obstacle by refusing to let them go. Some reports said Hezbollah leaders cited Reagan's tough speech in Chicago Heights last Friday, as the hostage release was falling into place at Assad's direction, as justification for their refusal.

White House sources said Reagan's remarks had actually been drafted two days earlier, and were generated by a concern that the hijackers might think the TWA hostages had become "more valuable" because Reagan was meeting with their Illinois-based family members. At the time Reagan spoke, his aides were unaware that a breakthrough was imminent.