SECRETARY OF STATE Cyrus Vance, and his press party, seemed glum when he finished his 11 days in Mideast. With some reason. As was amply publicized, the Israelis refused to yield on the two key demands the Carter administration is making of them: withdrawal from occupied territory and acceptance of a Palestine homeland. And, as should have been better noted, Arab leaders apparently did not go beyond their previous inadequate circumlocutions concerning their readiness for those contacts with Israel that both Israel and Mr. Carter rightly regard as the essence of peace; President Sadat of Egypt went in reverse, backing off an earlier hint that he would consider diplomatic and trade ties five years after "peace."

So it seems that the administration won't succeed in reconvening the Geneva conference in October, as it had hoped, or in opening early talks in some other diplomatic context, as Israel had suggested. Routine continuing contacts as the annual General Assembly meeting in the fall are all that's expected now.

But we're not as gloomy about all this as some. Something important and positive happened on the Vance mission: The results of the great pressure being put on the Palestine Liberation Organization by settlement-minded Arab states started to show. The PLO is still anathema to most Israelis. But it seems to have gotten itself pretty much over the hurdle of accepting the language referring to Israel in United Nations Resolution 242: "respect for and acknowledgement of the sovereignty, territorial integrity and political independence of every state in the area and their right to live in peace within secure and recognized boundaries . . ." That resolution's lack of reference to Palestinians other than as "refugees" apparently has kept the PLO from going further. But the matter is under intense debate within the PLO. The organization's stirring, we suggest, is a major event, more important than any speedup or delay on the road to Geneva.

Perhaps the PLO will ease the Israeli government's discomfiture by reaffirming its old hard time. The fact is, PLO movement toward moderation brings nearer the day of a U.S.-Israeli showdown. Washington cannot in honor urge Israel to have open dealings with an organization that does not accept the international consensus, embodied in 242, supporting a Mideast settlement. But it can urge Israel to deal with a PLO that, formally and unconditionally, accepts that consensus. The PLO's insistence in its "convenant" that Israel is "null and void" is obnoxious and genuinely disquieting but not necessarily crippling to Israeli security, especially given Israel's great strength, and not necessarily crippling to future negotiation prospects. If the PLO keeps its covenant intact, the Israelis can fairly intensify their demands for other Palestinian concessions.

We find it neither surprising nor disturbing that Israel has not already opened its arms to the PLO. By standing fast, Israel has contributed materially to the climate that led the Arab states to decide to put the heat on the PLO. Nor do we question the U.S. pledge, given by the previous administration and confirmed by the current one, not to force Israel to deal with the PLO unless the organization accepts 242 (and a companion resolution, 338) and grants Israel's right to exist. That pledge expresses the American commitment to Israel. It expedited the disengagement agreements of 1975. And it, too, has contributed to the pressure on the PLO.

At a certain point, however, if the organization with the best claim to represent the Palestinians meets the test of international responsibility, Israel will have to respond. That point, we believe, is some degrees closer than it was before Mr. Vance's Mideast mission.