Secretary of State Cyrus Vance arrived in Israel last night to begin a new American effort to break the deadlock in Egyptian-Israeli peace talks.

Vance is known to have had serious doubts about making the trip because of the lack of signs that he will emerge with anything new from Prime Minister Menachem Begin or from Egyptian President Anwar Sadat. President Carter personally decided to send Vance on the essentially exploratory mission.

The normally low-key Vance was even more cautious than usual in discussing his hopes for the trip with reporters before he landed at Ben Gurion Airport, where he was welcomed by Foreign Minister Moshe Dayan.

Vance said aboard his Air Force jetliner that there had been no indications from either side of new flexibility since Sadat announced last Sunday that he would not resume direct talks until the Israelis agreed in principle to complete withdrawal from Arab territories occupied in 1967.

"I decided that it was necessary for me to come out to the Middle East and to meet with the two heads of government to discuss their evaluations of where we stood, and to seek their views on how we could get negotiations resumed," Vance told reporters.

In response to questions, he indicated that the Carter administration was not insisting on resumption of face-to-face talks as the only way of keeping alive the peace initiative launched by Sadat's visit to Jerusalem last November. Vance said he had "an open mind" on the format of Egyptian-Israeli contacts, and that he "would be flexible" about staying in the Mideast if his talks produced an unexpected change in the deadlock.

Dayan struck a conciliatory note in welcoming Vance, saying that "in order to get a settlement everyone, every party, has to make compromises and concessions . . . You are probably about the only one who can now push forward the stuck negotiations."

Vance was to meet Begin for talks this morning before meeting with the Israeli Cabinet and then lunching with Dayan. An afternoon round of talks with Begin follows. Vance flies to Alexandria to see Sadat tomorrow and expects to return to Washington Wednesday evening.

Vance's arrival was preceded Friday night by an evident attempt by Dayan to soften Begin's blunt rejection last week of Sadat's request for a "goodwill gesture" to push negotiations forward. In a television interview, Dayan said that Israel was willing to negotiate the return to Egyptian civilian control of the Sinai capital of EL Arish before a final peace agreement is reached.

Sadat had made an informal request for the unilateral return of El Arish, a coastal Sinai town, during talks last month with Israeli Defense Minister Ezer Weizman in Vienna. Sadat suggested the next round of talks could be held at El Arish and at Mount Sinai, site of the Greek Orthodox monastary of St. Catherine's, which he also wanted returned to Egypt now.

The Israeli Cabinet last month rejected the proposal, which Begin publicly announced and then ridiculed by saying, "nobody can get something for nothing, and this is going to be the policy of Israel."

Reports reaching Jerusalem and Washington depict Sadat as having exploded with anger at Begin's remark and his making the proposal public. It is believed to have led to Sadat's expulsion of the nine-man military negotiating committee and the cancellation of direct talks now.

The trip involves Vance in assessing Israeli and Egyptian strategies and what the American response to them administration as having expressed a new willingness in the past month to discuss an eventual compromise on the West Bank and Gaza Strip territories after a transitional period, but not to have made any fundamental changes in position.

The administration was encouraged by other signs that emerged from the meeting of Egyptian, Israeli and American foreign ministers at Leeds Castle in England July 18 and 19.Public statements indicated that there had been discussion of the possibility, at least transitionally, of an Israeli military pressence remaining in the West Bank and Gaza as a result of an agreement. There was also discussion of a local Palestinian government for a transitional period.

The Carter administration sees its role as stimulating the direct talks and then trying to bridge the wide gaps that separate the Egyptian and Israeli position on these and other points. But Sadat's refusal last Sunday to go through with a new foreign ministers' meeting that Vance felt had been agreed upon at Leeds has pushed the United States back to trying to get agreement from both sides that there is enough common ground to hold talks at all.

Sadat says there is not, insisting that Israel accept the principle of total withdrawal before talks can be resumed. Israel now indicates it is willing to discuss the possibility of territorial compromise on the West Bank and Gaza, but that such talks could only grow out of a five-year interim period of local autonomy and continuing Israeli military presence in the territories.