Israels Foreign Minister Moshe Dayan is urging his government to accept a new American formula for the coming Middle east conference in Geneva, reliable sources reported here today.

According to the formula, the Palestinians would be represented in an all-Arab delegation at the formal opening of the conference, but immediately afterward the conference would split into committees in which Israel would negotiate with each Arab state separately.

It is not clear in this proposal where the Palestinians would fit in in the second stage of the talks. The Israelis would like them to become part of the Jordanian delegation.

Ephraim Evron, the director general of the Foreign Ministry who was at the talks in Washington this week, has been sent by Dayan back to Jerusalem to attend the weekly meeting of Prime Minister Begin's Cabinet on Sunday. Evron was expected to brief the Cabinet and provide them with clarifications and details.

According to the sources, the ideas presented to Dayan Monday by Carter and Secretary of State Cyrus Vance were also suggested to Egypt's Foreign Minister Ismail Fahmi, who visited Washington two days later.

The very fact that Begin has decided to take up the U.S. proposal at the Sunday meeting is seen here as a step forward. Observers here do not exclude the possiblity of Israel's demanding further clarifications on the suggested procedure.

If the government accepts the proposal. Dayan may strive for further progress in preparing the ground for the Geneva conference Monday, when he is to meet with Vance again. Reports reaching here from Washington do not exclude the possibility of another meeting between Dayan and President Carter.

The first hint that the United States is seeking a formula to make the Palestinians' participation at the Geneva conference possible came last week, before Dayan arrived in Washington.

It was reported then that Vance saw two possible ways in which the Palestinians could be present without clashing with Israel's determined opposition to negotiate with any delegation officially representing the Palestine Liberation Organization.

Vance is said to have suggested that the Palestinians be represented either in an all-Arab delegation or as part of the Jordanian delegation. He exprassed preference for the first possibility.

Israel, however, rejected the idea. As Dayan explained last week before leaving for Europe, there is always a danger in such a delegation that the extremists will dominate and the moderates will not be heard. Thus the PLO, whose positions are less compromising then those of Arab states, could obstruct, a settlement.

The reported new American formula overcomes this difficulty be assigning the all-Arab delegation only a formal role, asserting that the Palestinians are a party to any Middle East peace settlement but at the same time enabling Israel to negotiate without preconditions with each of the Arab states before confronting the thorny issue of the Palestinian homeland.

The compromise formula also skirts the issue of who will represent the Palestinians in the all-Arab delegation. Both the United States and Israel are said to prefer elected mayors from the West Bank and Gaza. Even if they are known as PLO sympathizers, they would be in Geneva as representatives of the indigenous West Bank population rather than as emissaries of any political movement.

The reported American compromise formula was bitterly attacked tonight by former Prime Minister Yitzhak Rabin. In a television interview Rabin, one of the leaders of the opposition Labor Party, warned against an ostensibly procedural solution such as an all-Arab delegation at the opening session of the conference.

"In such circumstances procedure can easily become substance, and the Arab position will be dictated by the most extreme members of its delegation." Rabin said.

He also severely criticized the Begin government for seeking a Geneva conference at this time and for aiming at an overall settlement without reaching a prior agreement with the U.S. government on future borders.

"The present policy will lead to an inevitable impasse and a breakdown of the movement toward peace set in motion four years ago," Rabin said.