With Secretary of State Cyrus Vance looking on, Egypt and Israel formally begin trying today to bridge the wide gap that separates their views on self-government for Palestinians in the Israeli-occupied territories.

In brief ceremonies at a university hall in this southern Israeli city, ministers from the two countries made clear that hopes of expanding the Egyptian-Israeli peace treaty into a settlement of the Palestinian question will require an effort more difficult and time-consuming than the six months of bargaining that went into the peace accord.

Egyptian Defense Minister Hassan Ali, speaking for President Anwar Sadat's government, declared that the Palestinian people alone can determine their fate and that Israeli settlements in the occupied West Bank and Gaza Strip are illegal.

The leader of the Israeli delegation, Interior Minister Josef Burg, responded that "autonomy does not and cannot imply sovereignty.

Reasserting Israel's stand that it never will agree to establishment of a Palestinian state, Burg charged that such a state would be a "a knife to the jugular vein of Israel."

Earlier, in a simple and stiffly formal ceremony lasting barely 15 minutes, El Arish and a narrow swath of the Sinai Peninsula extending Westward almost to the Suez Canl reverted to Egypt after 12 years of Israeli control.

At a dusty military parade ground on the outskirts of town, one senior Israeli military official, southern district commander Maj. Gen. Dan Shomron handed over El Arish in a correct but unemotional ceremony to Egyptian Gen. Mohammed Hussein Shawkat, who will assume command as governor of the district.

"We did not conquer this area out of a desire for expansion [but] rather by the necessity of a war situation which threatened our very survival," Shomron said. The turnover, he added, represents "the clearest and most sincere expression of our desire for peace."

As the military bands played the Isaeli and the Egyptian national anthems, Israel's flag was lowered. As it descended, hundreds of El Arish Arabs watching from behind a wire fence surged forward, cheering loudly and shouting nationalist slogans. They were pushed back by Israeli and Egyptian soldiers lining the field.

Immediately after the ceremony, the Israeli honor guard and a handful of remaining Israeli occupation officers immediately boarded trucks and automobiles and headed for the El Arish airstrip just outside town.

On Sunday, Vance, Egyptian President Anwar Sadat and Israeli Prime Minister Menachem Begin will reenact the turnover ceremony following what is expected to be a tumultous welcome Saturday for Sadat.

Despite the glaring differences between the Egyptians and the Israelis, at the opening of the talks in Beersheba, U.S. officials continued to express optimism that the talks can be made to work in a way that will both ease Israel's fears for its security and satisfy the Arab world's demands that the 1.1 million Palestinians in the occupied territories be able to shape their own future.

Vance, who emphasized both points equally, also underscored his expectation that the U.S.-mediated talks will take a long time. Noting that "the range of issues is far too complex to be resolved all at once," he added: "The only realistic approach is to establish a transitional period during which the decisions that need to be made can be dealt with in a measured and logical way."

With the return of El Arish and Sunday's planned ceremonies, the formal opening of negotiations on a self-governing mechanism for the West Bank and Gaza look place against a background that had been intended to demonstrate the completion of one phase in the Middle East peace process and the start of another.

Yet none of the carefully staged symbolic events has been able to dispel the clouds hovering over the peace process-Israel and Egypt's hard-line bargaining positions, the failure of other Arab countries to join the negotiations and continuing squabbles between the Sadat and Begin governments.

These differences were highlighted by the last-minute refusal of Egyptian Prime Minister Mustafa Khalil to come to today's ceremonies here. Instead, Khalil, who is to head the Egyptian negotiating team, cited "procedural problems" and sent Hassan Ali in his place.

Reliable sources said the Egyptians were miffed by Begin's refusal in the past to deal with Khalil as someone of comparable rank. The sources added, however, that Khalil will head the Egyptian side when the second round of talks begin in Alexandria, Egypt, on June 6.

Vance, who had conferred earlier in the week with both Begin and Sadat, flew to Israel from Cairo this morning for a luncheon meeting with Burg. The two then traveled by helicopter to a military airfield near this pioneer town that has been built out of the Negev Desert south of Jerusalem.

There, together with other Israeli Cabinet officials, they went through what has become an increasingly familiar sight in Israel since the start of the rapproachment process-the arrival amid military honors of an Egyptian jetliner bearing the Egyptian negotiating delegation.

Then, the entire group went to the university, where, in the spacious atrium of a starkly modernistic classroom building, Burg formally convened the first session of the talks agreed to by the two countries under President Carter's mediation. Except for a brief closed session dealing with procedural questions, the meeting today was given over to the speeches of the three delegation heads.

Hassan Ali, who spoke first, reaffirmed his country's hopes for a broadening of the peace. In a careful bow to Arab world sensibilities, howevver, he added that Egypt expects autonomy to mean an end to Israeli rule in the occupied territories and the right of the Palestinians to choose their own future.

"We are not here to determine the fate of the Palestinian people," he said. "Only they can do that." The Egyptian minister then spelled out additional demands-the return of East Jerusalem to Arab rule, the branding of Israeli settlements in the occupied territories as illegal and recognition that any Israeli laws or steps to change the status of the disputed areas "have no validity."

Burg, in response, said the Begin government believes "that the Palestinian Arabs should and must conduct their own daily lives for themselves and by themselves." But he then made clear that Israel's bargaining position is designed to perpetuate Israeli control over all but the most perfunctory of governmental functions in the occupied territories.

"What I must make clear and what must be understood from the outset is that autonomy does not find and cannot imply sovereignty," he said. "Israel will not agree and indeed totally rejects the establishment of a Palestinian state. It would be a mortal danger to Israel and a grave peril to the whole of the free world."

When Vance spoke, he took the position that the United States-while fully aware that the two sides are starting out far apart-believes their differences can be resolved through hard work, a spirit of compromise and good will.

Calling for a gradual, step-by-step process over the coming months, he laid equal stress on the need for "genuine acceptance" of Israel's right "to live in peace and security" and "the legitimate rights of the Palestinian people." He also expressed regret at the absence of such key Arab countries as Jordan from the talks and said, "We want to make it clear that the invitation for them to join us remains open."

Vance caused a stir within the Israeli delegation by saying there was a need to deal with the problems of Palestinians "living outside the West Bank and Gaza" in such countries as Lebanon and Syria.

After consultating with U.S. Embassy officials here, he dropped a sentence from his text that said, "they too must have a means of political expression and fulfillment." He did say, however, "they too must know that an accepted and respected place exists for them within the international community."

Under agreements made when the peace treaty was signed in March, the autonomy talks have a target date for completion within one year. Robert Strauss, the presidential trouble-shooter who will take over as chief U.S. negotiator, has refused to say whether that date can be met.

Strauss is scheduled to make an initial get-acquainted trip to the questions today, Vance said Strauss hopes to begin taking part in the negotiations sometime in August. CAPTION:

Picture, Arab residents of El Arish scuffle with Israeli soldiers during transfer ceremony. AP; Map, no caption, By Dave Cook-The Washington Post