Secretary of State George P. Shultz's unsuccessful mission to the Middle East this week brought U.S. initiatives in the region to an apparent dead end and dramatizes this fact in a way that could damage the Reagan administration's standing at home and abroad.

The administration is now searching for a new basis for U.S. policy in the Middle East before Lebanese President Amin Gemayel and Israeli Prime Minister Menachem Begin come to Washington later this month.

It is not yet certain how the administration will deal with the political problem of raising and then dashing public expectations during the secretary of state's hastily arranged Mideast journey.

How and why Shultz decided to go to the Middle East has been subject of much speculation, intensified by the sudden way the trip was scheduled and announced. The secretary was out of the country on a long-planned trip to the Philippines, Thailand, India and Pakistan when the decision was made during his stop in Islamabad, Pakistan, a week ago today that he should go to the Middle East before returning home.

According to his aides, a Mideast stopover had been under consideration since before Shultz left Washington June 23.

The way his trip had been structured, Shultz was to fly over the troubled Mideast on his way home. Failure to stop there would have given an impression of indifference that could haunt U.S. policy later, especially if the unsettled situation in Lebanon should lead to new violence.

On the other hand, soundings in the area by U.S. special emissary Philip C. Habib and others had found little to suggest that a breakthrough in the growing Lebanese stalemate was likely. Some officials in the Shultz party argued that he should not go unless there was a substantial chance of success.

"The odds changed almost daily" on the question of whether to go or not to go, a Shultz aide said. In a meeting in New Delhi several days before the final decision the consensus was reported to have been opposed to a Mideast stopover because the diplomatic reports from the region were so negative. But by last weekend the odds had changed.

Officials said the crucial point was that the position of Syrian President Hafez Assad, in some respects the central actor in the current Mideast drama, was unknown or at least uncertain.

The arrangements worked out by Shultz in his April-May shuttle had the effect of giving Syria a veto over Israeli withdrawal from Lebanon, but Assad had been very uncooperative in a meeting with Shultz in Damascus May 7, and since then had refused a visit by Habib to discuss the Syrian position further.

It seemed unlikely from the public and diplomatic evidence at hand that Shultz would be able to convince Assad to withdraw his forces from Lebanon, an action that would trigger a parallel Israeli withdrawal, or even that Shultz would be able to persuade Assad to negotiate on the matter with the Lebanese government. Yet no one could be certain without a direct discussion with Assad.

Given the crucial role of Assad, U.S. aides said, Shultz thought it was so important to establish whether the Syrian leader was willing to maneuver that it was worth the risk of leaving Damascus empty-handed.

For this reason--and in order not to "fly over" the Mideast--Shultz decided in a "council of war" with his staff after dinner last Saturday that he should go, and cabled President Reagan for approval.

Robert C. McFarlane, deputy national security affairs adviser, telephoned Shultz in Islamabad late Saturday, Pakistan time, to say that Reagan had approved the trip. Several hours later, in late afternoon, Washington time, the White House announced the mission, in time for the Saturday night news and the Sunday papers but in the middle of the night Pakistan time. The announcement caught the Shultz party by surprise.

The White House was forced to say vaguely that Reagan had directed Shultz to "make an effort to stop in the Middle East" and to decline comment on his Mideast destinations because U.S. diplomats in the area were still in the process of working out plans for the visit with the governments concerned.

The State Department, following its usual practice, decided that it would be impolitic to visit Damascus without also stopping to see the leaders of friendlier governments in the area--Israel, Lebanon and Saudi Arabia--and stopping off after this round of discussions to brief the leaders of Jordan and Egypt on the way home.

From the first confirmation of the visit, Shultz described his mission as primarily that of fact-finding, and he sought to discourage any idea of achieving a breakthrough. But the sudden trip of the U.S. secretary of state seized world attention anyway, especially on a summer holiday weekend when there was little other news, and gave rise to intense speculation that Shultz might be able to break the Lebanon deadlock.

As it turned out, Assad refused to budge in nearly five hours of conversation with Shultz Wednesday in Damascus, agreeing only to continue the dialogue in a U.S.-Syrian "working group" from which little is expected.

By this time, Shultz had been buffeted by unhappy tidings in his stops in Saudi Arabia and Lebanon, and Syria's adamant stand reinforced the difficulties with Israel, his next stop.

In an unexpected way, Shultz's visits seemed to bring to a head the cross-purposes, troubles and worries of America's regional partners that had been building since his last visit two months earlier. The trip not only confirmed the frustration of U.S. policy in the Middle East, but also dramatized it.

Shultz, who joined the administration a year ago in the midst of the turmoil of Israel's invasion of Lebanon, an event that contributed to the demise of his predecessor, Alexander M. Haig Jr., has sponsored two major drives by U.S. policy in the Middle East.

One of these, announced by Reagan Sept. 1, was to advocate a broad settlement of the Palestinian issue by bringing King Hussein of Jordan, with some form of Palestinian backing, into the negotiations on the future of the Israeli-occupied West Bank and Gaza Strip.

After months of jockeying, this U.S. drive was stopped cold April 10, when, after a rejection by Palestine Liberation Organization chairman Yasser Arafat, Hussein announced he could not participate in the negotiations.

The other major thrust of policy, related but technically separate, was the U.S. drive to negotiate the withdrawal of Israeli troops and other foreign forces from Lebanon.

After months of delay, Shultz in May conducted a 17-day Mideast shuttle mission resulting in an Israeli-Lebanese accord for the pullout of Israeli forces. However, Israel insisted--and the United States agreed in a still-secret "side letter"--that Israeli troops would not have to depart until there was a simultaneous withdrawal by Syrian and PLO forces.

Syria has played a key role in frustrating U.S. policy in both cases. In the first instance, Syria pressured Arafat not to acquiesce in U.S.-backed negotiations on the West Bank, and later encouraged a revolt against Arafat's leadership for even considering such an arrangement.

In the case of Lebanon, Syria denounced the Israeli-Lebanese agreement and refused to withdraw its troops in parallel with the negotiated Israeli pullout. Under the U.S.-accepted arrangements, this means that Israel is not obligated to withdraw.

Part of the problem, in the view of regional experts, has been a U.S. insensitivity to Syria and its concerns during the Reagan administration, which is uncomfortable dealing with a radical state. A related factor has been the growing ties of intimacy and dependency between Syria and the Soviet Union, especially following Syrian military setbacks in last summer's Lebanese battles.

Still another factor has been the unwillingness or inability of Saudi Arabia to cast its weight strongly behind the U.S. proposals for the region. Washington had hoped that the Saudis would help convince Syria to withdraw its forces from Lebanon, despite Assad's dislike of the Israeli-Lebanese arrangement, but instead Assad seems increasingly to have brought the Saudis around to his viewpoint.

These are times of trial for Saudi Arabia on a variety of fronts: the oil glut and unresolved battles among oil producers have diminished the kingdom's leverage and sharply cut its income; the Syrian-backed attacks on Arafat, the Saudis' close ally in the PLO, threaten further radicalization of Palestinian leadership; the lack of success of the U.S. policy, to which the Saudi stance contributed, has created new uncertainty among the Saudis about their American connection.

There is a perception throughout the region that the United States, having sponsored the Sept. 1 "Reagan plan" for regional negotiations and the talks for Israeli withdrawal from Lebanon, has done too little, too late to push them to fruition.

Even some senior U.S. diplomats in the area said that, in retrospect, U.S. chances for success would have been greater if it had pushed harder on both the Palestinian and Lebanese fronts last fall and in the early months of this year, rather than waiting for regional forces to jell and events to take their course.

At the heart of this difficulty is U.S. relations with Israel, which was never happy about the Reagan plan and which laid down tough conditions for withdrawal of its forces from Lebanon. Until Ariel Sharon was fired as Israeli defense minister in February, U.S. policy makers thought there was little chance for a negotiated Israeli pullout from Lebanon without a major showdown with the Begin government, a showdown Washington was reluctant to undertake.

As of today, Syria refuses to withdraw from Lebanon; Israel wants to make a partial withdrawal, or redeployment, of its forces in Lebanon to more defensible positions for the long haul; the Lebanese government strongly opposes any Israeli pullback that risks renewal of civil war or implies a lengthy occupation of part of Lebanon. Unless a sense of movement toward peace in the region can be reestablished, many experts fear a slide could begin in the opposite direction, toward greater instability and bloodshed.

Lebanon, which has been a cockpit of conflict for most of a decade, could again become a battleground. This time, the stakes for Washington are higher than ever, with 1,800 Marines on duty there, a U.S.-aided government in power and a U.S.-sponsored agreement for withdrawal of Israeli forces mired in a stalemate that Shultz was unable to shake.