The Middle East peace talks ran into serious new problems yesterday, with the United States and Israel engaging in a sharp public dispute over Israel's decision to enlarge its settlements on the occupied West Bank of the Jorden River.

The Carter administration fears that the Israeli decision will be taken as a deliberate provocation in the Arab world. That, in turn, could force Egypt to pull back from what had been regarded as imminent agreement on an Egyptian-Israeli peace treaty.

Concern about the possible effects on the treaty negotiations caused Secretary of State Cyrus R. Vance to issue an especially sharp statement calling the Israel plan, agreed upon Wednesday, "a very serious matter." He added, "We are deeply disturbed by it."

The State Department also revealed that President Carter had sent a message to Israeli Prime Minister Menachem Begin expressing U.S. concern and asking for an explanation.

Underscoring the U.S. fear about new snags was an announcement by President Anwar Sadat's government in Cairo that Egypt is considering recalling its chief negotiators from Washington for consultations.

Egyptian officials said the possible recall would be partly to discuss changes Egypt wants to make in the draft treaty agreed on by the negotiators here over the weekend. But the Egyptians also made clear that their contemplated action would be in part in response to the Israeli settlement decision.

But, despite the disquiet being expressed in Washington and Cairo, the first Israeli response was one of defiance. Foreign Minister Moshe Dayan, who returned to Washington last night after consultations in Jerusalem, said Israeli settlers on the West Bank would remain there whether or not the United States and Egypt approve.

Dayan charged that the United States was to blame for the dispute. In particular, he singled out statements made by Harold Saunders, assistant secretary of state for near eastern affairs, as having provoked serious concern within Israel.

During a recent visit to the West Bank, Saunders told Palestinians there that the staus of Israelis remaining in the settlements would be subject to negotiation.

That, Dayan said, forced the Israeli cabinet to take action "to assure the people they could stay - that we do not intend to move [from the West Bank].

"There is no question about it," Dayan said on his arrival at Andrews Air Force Base. "We don't dream about removing or dismantling one single Israeli settlement. I want to make it clear we don't feel we have to apologize about it."

These developments highlighted a day that saw sources connected with the U.S. medicated negotiations hastily pulling back from their optimistic predictions of recent days that the Egyptian-Israeli peace treaty would be wrapped up and ready for initialing by this weekend or early next week.

Also serving to cloud the outlook for the talks were reports yesterday that Israel is considering a plan to move the offices of the prime minister and foreign minister to east Jerusalem to reassert Israeli sovereignty over that part of the city, which it has occupied since 1967.

In addition, Syria and Iraq, ending a long and bitter feud, signed a "charter for joint national action" to fight against the Egyptian-Israeli rapproachment launched last month at the Camp David summit.

The Syrian-Iraqi accord could have an important polarization in the Arab world and could hobble U.S. efforts to coax other key Arab countries, like Jordan and Saudi Arabia, into supporting the drive for a negotiated settlement of the tensions between Israel and its Arab neighbors.

However, the matter causing the greatest concern and preoccupation within the U.S. government was Israel's announced intention to "thicken" its existing West Bank settlements by building new housing there.

U.S. sources said privately they were unsure whether Israel intends to move substantial new numbers of settlers to the West Bank, or whether the decision was a symbolic gesture designed to demonstrate Begin's independence of Washington and assuage members of his government who dislike parts of the draft treaty.

That question, the sources said, will probably be clarified when Dayan meets with Vance this morning and presents the revisions that the Israeli cabinet wants made in the draft treaty.

The West Bank actually is outside the scope of the treaty, which is to deal with the nature of peace between Israel and Egypt. However, Sadat sensitive to charges that he is ignoring larger Arab interests in making peace with Israel, has insisted that the treaty contain a reference to the need to solve the problem of the West Bank and its Palestinian inhabitants.

Israel, arguing that the Camp David accords called for the West Bank and the Israeli-occupied Gaza Strip to be dealt with separately, has contended that these areas should be left to later negotiations after the present talks are concluded satisfactorily.

Last Saturday, following a personal intervention by Carter, a compromise was reached. Although its precise details are still secret, the compromise reportedly centered on establishing a link to the West Bank issue in the treaty language that Israel could regard as not legally binding.

However, when Dayan took the draft treaty back to Jerusalem for approval the cabinet approved it, subject to changes tht reportedly will weaken the West Bank linkage language still further. Dayan is to present the changes sought by Israel to Vance today.

At the same time, the Egyptian government has indicated that it will seek changes aimed at strengthening the linkage. U.S. officials said yesterdy that they did not yet know the nature of the revisions being sought by the two sides, but they expressed concern that conflicting demands by Cairo and Jerusalem might undo the compromise achieved last Saturday.

As of last night, a tentative meeting scheduled for today by leaders of the three delegations had been left in doubt by uncertainties over whether Egypt will call its negotiators home for consultations.

Egyptian Prime Minister Mustafa Khali said in Cairo that the Sadat government was considering such a move. But George Sherman, spokesman for the talks here, said last night that the Egyptian delegation so far had received no instructions to return.

Given the continuing uncertainties over the linkage question, the Israeli decision on the settlements could become an especially irritating new problem.

The official U.S. position is that settlements on occupied territory are illegal. At Camp David, Carter and Begin reached an oral agreement to freeze new settlements on the West Bank, but they later disagreed about how long the ban would last.

Carter also agreed to some Israeli expansions of existing settlements, but he later made clear that it would involve only small numbers of people and be aimed mainly at reuniting families. U.S. sources said yesterday that the Israeli decision appears to involve much larger numbers than Carter had agreed to, and, they added, that was what prompted Vance's protest and Carter's request for an explanation.