The United States is paying a heavy price for a diplomatic failure that could contribute to the partition of Lebanon and end hope of reviving President Reagan's Middle East peace initiative.

Having deliberately kept Syria on the sidelines throughout the Israeli-Lebanese negotiations on the evacuation of Israeli forces from Lebanon, U.S. policy makers now seem to have no idea how to deal with Syria and no notion, even, of what President Hafez Assad's real objectives are in the present stalemate over the withdrawal agreement.

"The Sphinx has moved from Cairo to Damascus," commented one frustrated State Department official.

Meanwhile, Lebanese President Amin Gemayel told all who listened to him during his recent visit here that time is running out fast on his efforts to hold together his embattled government and a fragile national consensus favoring the May 17 accord, which has been made ineffective by Syria's refusal to withdraw its troops from Lebanon. Israel had made its pullout dependent upon the withdrawal of Syrian troops. NEWS ANALYSIS

Just as worrisome is the perceptible change in the Arab attitude as time passes and the Syrians keep up their aggressive campaign to undermine the initial support of the agreement--or at least passive acceptance of it--by the majority of moderate Arab states. Saudi Arabia, which first signaled what seemed to be a willingness to back the accord, is now expressing its "worry" that Lebanon has gone too far toward normalizing relations with Israel.

In a survey of Arab, Israeli and U.S. diplomats tracing the causes and effects of the failure of U.S. diplomacy in the Middle East, Washington Post correspondents have learned that a tough letter from Saudi Arabia's King Fahd to President Reagan in late June, warning that American inactivity was undermining Fahd's position at home and in the Arab world, was a major factor in Secretary of State George P. Shultz's spur-of-the-moment decision to visit the Middle East on his way back from a trip to Southeast Asia.

The Saudi monarch reportedly warned the administration that it would have to do a lot more to help sell the agreement and emphasized the mounting danger to both the U.S. position in the region and that of moderate Arab leaders if it did nothing to overcome Syrian opposition.

That warning carries ominous implications for a U.S. diplomatic strategy that depends on Saudi Arabia as its main lever to move Syria. Fahd warned U.S. officials that he has no more influence with Syria than Washington has with Tel Aviv.

"The Saudis say, 'We deal with Syria in the same delicate way you deal with Israel,' " said one U.S. official.

A high-ranking Lebanese official warned that if the present stalemate continues much longer, "for us it is tragic, for you disastrous."

The first step in the new quest for some understanding with Syria has already been taken with the replacement of the chief U.S. negotiator, Philip Habib, and his top assistant, Morris Draper, by a new team led by Robert C. McFarlane, deputy national security affairs adviser, who, unlike Habib, will at least not be persona non grata in Damascus.

Whether the current impasse could have been avoided by a more artful turn of diplomacy toward Syria is a subject of much debate among the U.S. diplomats who were involved. Many insist that Assad sent false signals that Syria would probably be willling to go along with the agreement.

Others say the danger signs were clearly visible to those willing to read them. These people are still surprised by what they see as the lack of understanding shown by Shultz, whose previous business experience in the Middle East was expected to give him special insight into how to deal with the Arabs. Instead, some trained observers in the field found him almost naive about the political dynamics of the Arab world.

"We thought that spending time in the Middle East meant that he had understood a good deal about the Middle East," said one middle-level U. S. official who asked not to be named. "We found out we were wrong."

What is clear from interviews with a score of officials here and in the Middle East is that Shultz, Habib and those above them ignored or discounted persistent warnings, from U.S. embassies in the area as well as from old Syrian hands in Washington, that strong Syrian opposition was likely. Thus American and Lebanese negotiators decided to provide the Syrian leadership with only the sketchiest information about the substance of the deal.

The United States, in fact, decided early on not to deal directly with the Syrians and left it up to the Lebanese to decide how much to pass on to them.

"We thought it best to let the Lebanese take the lead in how to deal with the Syrians," said Thomas Homan, State Department Middle East public affairs adviser. "Habib feared flights to Damascus would muddy the waters of the Lebanese-Israeli talks."

The Lebanese, in turn, decided they were not going to tell the Syrians very much about what was going on until it was all but over.

"Had we gone to discuss with them the agreement, they would have been very angry," said a senior Lebanese official. "Had we discussed it, they would have argued over every clause."

William Quandt, a Middle East expert at the Brookings Institution, remarked, "I think it was a problem for which there was no solution and so no one wanted to deal with it."

Yet the first private indication that Syria would not go along with the probable outlines of an Israeli-Lebanese accord came even before formal negotiations got under way last Dec. 28. A former U.S. official who saw Syrian Foreign Minister Abdel Halim Khaddam in mid-December and discussed the already known "minimum" Israeli demands said Khaddam immediately replied, "Of course we won't withdraw if these are the conditions."

His remarks were passed on to the State Department, but without effect.

The Syrians relayed their views to the Lebanese Jan. 15, when Khaddam told a Gemayel envoy that Lebanon "must not allow Israel to achieve any political gains while ensuring a withdrawal of the Israeli occupation forces from Lebanon."

That Syrian opposition was stiffening became clearer with each passing week. By early February, Habib was getting an increasingly cool reception on visits to Damascus, and Assad simply refused to see him any more.

In what appears to have been one of its first major miscalculations, the Reagan administration assigned Habib the task of getting Syrian, Israeli and Palestinian troops out of Lebanon, even though it was known in the State Department that Assad was distrustful of Habib and furious with him because of the American's role in negotiations during last summer's Israeli invasion of Lebanon.

Assad reportedly was convinced that Habib had double-crossed the Syrians by giving them assurances that Israel would observe a cease-fire negotiated by Habib and would not attack Syrian missile batteries in Lebanon's Bekaa Valley. Habib's assurances were delivered only a few hours before the Israelis destroyed the batteries.

Despite these many storm signals, there is no evidence the Reagan administration ever seriously considered using its influence with the Israeli government to get it to concentrate on the narrower issue of a troop withdrawal and dropping of its demands for a Lebanese commitment to the process of normalization.

"It was very hard for our policy makers to say Lebanon should not take a step toward normalization," recalled one official, particularly in light of the U.S. commitment to the Camp David process aimed at the making of peace between Israel and its Arab neighbors.

Nor was serious consideration given, apparently, to getting tough, as former president Jimmy Carter did after the Israeli invasion of Lebanon in June 1978, and telling Israel that its relationship with Washington would suffer seriously if it did not leave southern Lebanon. Former secretary of state Cyrus Vance briefed Shultz last summer on the earlier U.S. tactics.

The Reagan administration did press Israel to accept the general framework for a progressive normalization of relations with Lebanon rather than the immediate formal peace treaty Israel sought. Reagan also refused to invite Israeli Prime Minister Menachem Begin to Washington until an accord was signed, and during the final tough weeks of bargaining, administration officials backed their negotiators in their insistence that there should be no permanent Israeli military presence in southern Lebanon.

Even so, the difference between the final agreement and a proper "treaty" remains so thin in Israeli eyes that officials in Jerusalem regularly use the word "treaty" when referring to the accord.

Yet top State Department officials publicly declared that there were serious dangers in pushing Lebanon too far toward normalization of relations at the outset. In a speech last December to the Chicago Law Club, Deputy Secretary of State Kenneth W. Dam warned, "Progress toward more normal relations must be approached carefully lest it undermine Lebanon's credentials in the Arab world....Moreover, ill-timed or forced normalization may actually threaten Israel's security if it should lead to the breakdown of the Lebanese national consensus, thereby inviting the return of hostile forces."

Lebanese and U.S. officials tend to view the other party as primarily responsible for acceptance of the broader accord, verging in form and language on a formal peace treaty.

A high-ranking Lebanese official who accompanied Gemayel to the United States gave this version of the U.S. attitude after the Israeli government's first public statement last Oct. 10 of its terms, including a peace treaty, for leaving Lebanon:

"From the very beginning, the Americans really terrified us. They told us, 'They the Israelis want peace and normal relations.' We said, 'This is very difficult for Lebanon to do,' and they said, 'Can you tell us how else you are going to get them out?' "

But U.S. officials insist that whatever the initial Lebanese shock over Israeli demands, Lebanon's negotiators soon were ready to cross their own declared "red lines" and go much farther toward a peace treaty than even the Americans thought wise. Agreement on ending the "state of war" between the two nations and on Israeli establishment of a liaison office--the acknowledged embryo for an embassy--in the Beirut suburb of Baabda came early on in the talks, they insist.

One official noted that Lebanon's Sunni Moslem leaders, including Prime Minister Shafiq Wazzan, are still behind the accord.

In retrospect, many of those involved in the negotiations seem to feel the most serious error made by the United States was failing to make good use of the time between the evacuation of the last Palestinian guerrillas from Beirut on Sept. 1 and the start of negotiations on Dec. 28.

After the uproar in Israel over the degree of indirect Israeli involvement in the massacre of Palestinian civilians in the Sabra and Shatila refugee camps, Israel appeared to be in a weakened bargaining position, while Syria was still licking its wounds from the summer war and was even weaker than Israel.

Furthermore, Reagan's Sept. 1 Middle East speech, calling on Israel to accept an exchange of land for peace with its Arab neighbors, had been well received by most Arab leaders and American prestige was again on the rise.

In that reaction, however, was the seed of the impasse that has now developed.

"The top guys in the administration sat around saying how well they had done and did not really try to mount any follow-up diplomatic campaign to put it quickly into action," one observer said. This gave the Israelis more room to maneuver in Lebanon as well.

The initial American plan called for negotiations to begin in the first week of November, and it was calculated that they could be wrapped up by January or early February. Habib even predicted an accord before the year's end.

U.S. officials say the Israelis were primarily responsible for delaying the opening session for nearly two months by a variety of tactics, including secret direct talks with the Lebanese.

"What you needed was a fairly fast, uncomplicated agreement," said one close observer. "Timing was a crucial factor. What was possible in January and February wasn't possible in May and June. Psychologically the situation was changing and the Syrians were getting too strong. We should have demanded a much less detailed agreement and kept to the objective of getting Syria out."

Officials have belatedly recognized that the strategy of leaving Syria out of the picture began to fail quickly.

"We should have spent more time talking to the Syrians," remarked one official, who earlier was convinced that the Syrians would see that it was in their interests to pull out of Lebanon, if only to get Damascus out of range of Israeli artillery.

Now the United States finds itself dealing with a Syria that has been rearmed by the Soviet Union and Assad is in a strong military and political position domestically as well as in the Arab world.

U.S. officials insist there is no sign that Assad is in any hurry to open talks. For instance, the only item Syria has put on the agenda of the newly established U.S.-Syrian liaison committee, they say, is a date for Foreign Minister Khaddam's trip to Washington for talks during the U.N. General Assembly opening session in late September.

They also say they have had no indication that Assad is primarily interested in discussing the recovery of the Golan Heights, which Israel has been occupying since the 1967 war and has now virtually annexed.

The Syrian leader did not mention the Golan during five hours of talks with Shultz July 6, according to several officials.

"It was all Lebanon," one said.

One theory in policy-making circles is that Assad, as a minority Alawite Moslem faced with a conservative Sunni opposition, is more interested in issues affecting his personal security than in the Golan Heights. In this perspective, continued Syrian control over eastern Lebanon to block gun running and the passage of dissidents into Syria takes on a special importance.

Thus, the makings of a very cantankerous relationship between Damascus and Washington, if not a total stalemate, seem in place. Furthermore, insiders say the administration, embroiled in Central America, is looking for a "quick fix" for the problem while none is in sight.

"Remember, it took Henry Kissinger 33 trips to Damascus before Assad finally agreed to a disengagement agreement," remarked one official, referring to the former secretary of state's "shuttle diplomacy" to arrange the May 1974 accord separating Israeli and Syrian troops on the Golan Heights

"You have got to keep going and trying . . . ."