When Secretary of State Cyrus R. Vance undertook his Middle East mediating mission last week, he is known to have done so with reluctance and a feeling that he had only an outside chance of success.

Vance was keenly aware of the magnitude of the obstacles standing in the way of a peace treaty between Egypt and Israel. But the record of his five-day shuttle effort suggests that the Carter administration, in its handling of a high-risk deplomatic situation, managed to make Vance's taks all but impossible.

As a result, Vance not only failed to break the deadlock in the negotiations in time to meet today's Camp David agreed target date for a treaty, but also left the region with U.S.-Israeli relations badly strained and the talks so snarled that no one can predict when and how they can be put back on track.

Vance proceeded to the Middle East on the assumption that the realities of the situation required some difficult and painful concessions from Israel. In pursuing that goal, he first attempted to get them into as palatable a form as possible and then to urge them on the Israelis in the manner of a family doctor trying to coax a suspicious child inot swallowing medicine.

Yet, in an almost inexplicably strange counterpoint to Vance's conciliatory approach, a series of events began unfolding in Washington that had the effect of undercutting his efforts.

First, Senate Majority Leader Robert C. Byrd. (D-W.Va.) walked out of a meeting with President Carter to warn that future U.S. aid to Israel could be affected by Israeli policy on the sensitive question of putting settlements in the Israeli-occupied West Bank of the Jordan River.

Then, the president, in public remarks that also were echoed by other White House officials, praised Egyptian President Anwar Sadat, while pointedly hinting through his failure to mention Israeli Prime Minister Menachem Begin that Washington was becoming impatient with Israel's failure to make concessions.

In terms of what Vance was then trying to do in Jerusalem, the timing couldn't have been more disastrous. It touched off a storm of protest from Israeli political leaders and the press about unacceptable American pressure and literally ensured that Begin's cabinet would reject the proposals Vance brought.

In fact, the cabinet went even further, appending to its rejection of the proposals a slam at "the attitude and interpretation of the United States government." By the time Vance returned home. Israeli leaders were charging that the United States had tilted openly toward Egypt and was no longer playing the role of impartial mediator in the Middle East talks.

The result has been a muted, but unmistakable countering wave of irritation toward Isreal by U.S. officicals that includes charges about the Begin government deliberately misrepresenting the proposals conveyed to it by Vance. But, in the midst of the mutual recriminations, U.S. sources in Vance's party seemed unable to explain why Washington behaved with what some privately conceded was heavy-handed clumsiness.

In private, the sources admit that Vance did go to Israel with a position that could be described as tilted toward Sadat's position on the two key issues stalling the peace talks.

One involves Egytp's demand that the treaty be accompained by a time-table for separate negotiations on establishing Palestinian autonomy in the West Bank and Gaza Strip areas.

The other concerns Egypt's reluctance to accept an article in the draft treaty accepted by Israel that would give the accord precedence over Egypt's other treaties, including its mutual defense pacts with other Arab states.

In approaching thses issues, the sources said, the United States concluded that Egypt required assistance on both demands in order to protect it from charges that it is ignoring larger Arab interests in order to make a separate peace with Isarel. In particular, the sources added, these issues are important to win support for the treaty from Saudi Arabia.

In addition, the sources said, the United States also likes the idea of a West Bank Gaza timetable because it feels the Israelis have not really faced up to the need to grant Palestinian autonomy. By attaching the timetable linkage to the treaty with Egypt, the sources said, Washington feels Israel would be under greater pressure to negotiate the Palestinian issue quickly and in good faith.

However, Israel vehemently has resisted both Egyptian demands. On the Palestinian timetable question, the Israelis argue that the Camp David accords call for that issue to be negotitaed separately and that Egyptian demands for linking it to the peace treaty violate those accords and give Egypt a potential pretext for abrogating the peace treaty if the timetable isn't met.

The Israelis apply the smae objection to the dispute about the treaty's precedence. In the Israeli view, any suggestion that Egypt is not bound to remain at peace with Israel even in the event of an Israeli dispute with another Arab country would make the treaty meaningless.

In an effort to overcome the Israeli objections, Vance began his Middle East trip by spending three days in Cairo negotiating with Sadat to find ways of making the Egyptian demands more acceptable to Begin. Ultimately, they did agree on some U.S. suggestions for softening the original Egyptian positions.

These included a proposal to express the timetable on Palestinian autonomy in terms of nonbinding target dates rather than fixed deadlines. On the question of the treaty's precedence, Vance convinced Sadat to leave the treaty language unchanged and settle instead for an accompanying explanatory note stating a nation's right to self-defense under the United Nations charter.

Sadat also held out for some other concessions that the United States regards as minor and one additional piece of linkage that Vance knew would cause trouble in Israel. That involved an Egyptian insistence that, even after the oening of diplomatic relations with Israel, Sadat could refrain from sending an ambassador to Israel until Palestinian autonomy had been established in the occupied territories.

When Vance took these proposals to Israel, the U.S. sources said, he had no illusions that they would be accepted by Begin's cabinet. But, they added, he hoped to coax the Israelis into giving ground on the lesser issues and at least to agree to negotiate further on the timetable and treaty precedence questions.

Despite a hurried State Department effort to disassociate the executive branch from Byrd's remarks, Vance was greeted by charges of "U.S. blackmail" and a stiffening of the Israeli conviction that he had come, as one source put it, "to ram Sadat's proposals down their throats."

Begin also reportedly "went through the roof" when he heard about Sadat's proposal to tie an exchange of ambassadors to Palestinian autonomy. In fact, the sources said, Vance's reception in his first meeting with the cabinet was so hostile that immediately afterward he called Carter, and a decision was made - partly because of the Israeli reaction and partly because of the impending U.S. announcement on normalizing relations with China - to cancel the shuttle effort.

Even then , the sources said, Vance's delegation clung to a slim hope that the Israelis would hold the door open a crack to further negotiations on the Egyptian proposals.

But, that evaporated when the White House, its patience apparently exhausted, made public its annoyance at the Israelis. On Friday, when the cabinet met to formally reject the proposal, it did so with a veheemence that spurned even the more innocuous Egyptian ideas and added a slam at the United States for good measure.