Secretary of State Cyrus Vance and President Anwar Sadat, facing an expected Israeli rejection of new proposals for a peace treaty with Egypt, pledged tonight to continue the quest for peace in the Middle East.

The two met here as Vance ended his shuttle mission between Cairo and Jerusalem by reporting to Sadat on the hostile reception given the proposals by Israeli Prime Minister Menachem Begin's government.

Vance, who announced yesterday in Jerusalem that he was breking off his mediation effort, returns to Washington early Friday. His plane will leave Cairo about the same time Begin's Cabinet begins a formal meeting that seems certain to result in rejection of the latest Egyptian ideas for Breaking the deadlocked treaty negotiations.

Both Vance and Sadat said tonight that, until the Israeli Cabinet acts,they would make no public statements on Israel's attitude or on the possible next steps that might be taken to get the two-month-old treaty talks moving toward a successful conclusion.

They refused to talk to reporters except for brief, general statements as Sadat saw Vance to hos car following their session at a villa outside Cairo.

Vance said, "We will continue in the pursuit of peace." Sadat, asked if he thought an Egyptian-Israeli peace treaty will still be achieved, replied: "Yes, sooner or later whatever the consequences are."

However, the net result of Vance's five days in the Middle East have made clear not only that the treaty negotiations will be incomplete by the Sunday deadline set at the Camp David summit meeting, but also that they are still snarled on the same key issues blocking progress for more than a month.

In three days of talks here earlier this week, Vance and Sadat agreed on some compromise ideas for dealing with these problems. But when Vance took them to Israel Wednesday, he found the reaction so negative that, following a talk by telephone with President Carter, he concluded that there was no point in continuing the shuttle effort at this time.

But while Vance was on his way here late this afternoon, a high-ranking aource aboard the secretary's plane denied that the abrupt cancellation of his mission was intended to put pressure of Israel or underscore Washington's unhappiness with Begin's refusal to be more flexible.

The source said Vance had been summoned home because of urgent matters that he has to discuss with the president. Although the source would, he added that it probably would be made public by the White House Friday night.

In addition, the source added, the business did not involve the Middle East and is not related to a crisis.

Still, despite the denials of U.S. pressure and displeasure with Israel, there is little doubt that the Carter administration feels Sadat has made a strong effort to compromise his differences with the Israelis and that it now is up to Begin to show considerably greater flexibility.

While the newest impasse has revived a lot of speculation here and in Israel about a possible new summit conference with Carter again mekiating between Sadat and Begin, U.S. sources said categorically that no firm ideas have even been discussed about possible next steps.

First, the sources said, the specific response of the Israeli Cabinet has to become known. After that, the sources added, there is likely to be a brief pause, probably until after the Christmas holidays, while all three governments consider ways of getting the talks back on track.

but the sources also cautioned that Washington does not want too much delay because of fear that support for the treaty will begin to erode in Egypt and Israel. That, they said, was why Vance made the trip here even though he was aware that the chances of a breakthrough were slim.

The sources said the proposals Vance took to Israel on Sadat's behalf covered roughly five areas of disagreement. Some, they added, involved relatively minor issues that are not expected to cause much difficulty.

But, they conceded, at least two touch sensitive nerves in Israel and provoked very hostile reaction when Vance presented them to Begin's Cabinet yesterday and today.

One involves Egypt's continuing demand for the treaty to be accompanied by a timetable covering separate negotiations on establishing Plaestinian autonomy in the Israel-occupied West Bank and Gaza Strip. Under prodding from Vance, the sources said, Sadat softened the demand to having the timetable set target dates rather than compulsory deadlines.

However, while Israel has stated its willingness to negotiate the Palestinian issue, it was clear in Jerusalem that powerful forces in the Begin government are unwilling to accept any timetable. The Israelis feel that if Palestinian autonomy is not achieved by the specified timetable dates, Egypt could use that as a reason for renouncing the treaty.

In addition, it became known while Vance was in Israel that Egypt, while willing to open diplomatic relations with Israel, wants to postpone an exchange of ambassadors until Palestinian automony is actually established. The sources said the Israelis consider that totally unacceptable.