Secretary of State George P. Shultz leaves for the Middle East tonight on what many say is an elusive quest for a negotiating breakthrough that will ease tensions in Israel's strife-torn Palestinian territories and open the way to a settlement of the Arab-Israeli conflict.

No one seems more aware of the difficult task before him than Shultz himself. At a Brussels news conference while en route here from Moscow yesterday, he acknowledged: "It's going to be tough. I don't think many people give me much chance and that's the drift of their questions everywhere: intense skepticism."

But Shultz, usually poker-faced, offered a rare insight into his emotions about his high-risk Middle East mission:

"I believe that if there are chances, even if the chances are small, it's worthwhile trying. You can't be too afraid of failing. Suppose I go and don't succeed? What am I saving myself for? So we'll try, and people want to have the U.S. come, and maybe we'll get somewhere."

Impelling him are almost three months of turmoil in the West Bank and Gaza Strip in which more than 50 Arabs have died, intense pressure from pro-Israeli and pro-Palestinian groups here, and appeals by Arab heads of state and Israeli leaders for the United States to reengage itself in the peace process.

Many analysts, including a number of U.S. officials who are reluctant to say so publicly, see little that Shultz can do beyond creating an illusion of movement in a process whose outcome hinges on the interplay between Israel and the Palestinians.

"Nobody agrees with the American proposals but everybody wants a process," said Martin Indyk, executive director of the Washington Institute for Near East Policy.

Many Middle East analysts say the violence has changed permanently the relative Palestinian docility that held sway in the West Bank and Gaza Strip since Israel occupied them in 1967.

As a result, they argue, the Israeli public and government, almost evenly divided between those who want to keep the territories and those who would "trade land for peace," cannot indefinitely defer a decision without risking greatly increased confrontation with the frustrated young Palestinians who sparked the violence of recent weeks.

These observers foresee escalation of tensions that, in the short run at least, is likely to strengthen the hand of Israeli hard-liners advocating tougher repressive measures.

But, proponents of this theory argue, such tactics are unlikely to work against the raised expectations of the 1.5 million Palestinians in the territories.

In the end, they predict that Israel, a state born out of the Holocaust of World War II, will be unable to avoid a choice between letting the territories go or trying to hold on to them, either through mass expulsions or repression too brutal for a state with Israel's history and democratic ideals to contemplate.

Those U.S. officials and others who hold this view believe it will take two to five years for the process to reach that stage. In the meantime, they contend, many of the ideas that have dominated the Mideast peace debate in the past have lost much of their relevance because events have overtaken them.

These include arguments about whether Israel should deal with the Palestine Liberation Organization or only Jordan, whether negotiations between Israel and an Arab partner should be direct or part of an international conference, and whether there should be a period of limited autonomy in the territories before their final status is decided.

Whether that is the case should start to become clear almost as soon as Shultz embarks on his mission, carrying to the region the outlines of a U.S. plan he has described as a "blend" of these older ideas and some new variations that he hopes just might break the logjam.

Shultz basically is seeking to turn the negotiating process upside down by starting informal talks first on the "substance" of the dispute rather than dwelling on the "procedures" of how to hold an international conference.

This is partly because Israeli Prime Minister Yitzhak Shamir has steadfastly resisted Arab demands for a U.N.-sponsored conference attended by the Soviet Union where international pressures on Israel to yield "land for peace" likely would be intense.

But Israeli and U.S. sources say there is another reason for Shultz's fixation on "substance."

They say that last month, during a State Department review of the failings and accomplishments of the 1978 Camp David accords, Shultz discovered that Egyptian and Israeli negotiators, before breaking off talks in late 1981, had made considerable progress toward agreement on the rudiments of limited autonomy for the territories.

"Shultz found what had been agreed to under the Camp David talks breath-taking," one Israeli source said.

Shultz's revived interest in the now moribund Camp David talks apparently stems partly as well from Shamir's stated willingness to discuss changes in those accords.

But Egyptian President Hosni Mubarak and Jordan's King Hussein insist that Camp David can no longer serve as a framework for negotiations and that whatever "substance" Shultz has in mind must be discussed under the umbrella of an international conference.

These two opposing views toward new peace talks, plus the Palestinian demand for self-determination, are apparently what Shultz had in mind when he referred yesterday to the "extreme solutions" he will have to reconcile to get new peace talks going again.