An American-Israeli accord on a formula to start new peace talks on the Middle East was worked out early today, but Syria is reported opposed and other Arab nations also may balk at the terms.

Nevertheless, with this measure of agreement, the Carter administration and Israel, along with Israel's supporters in the United States, pulled back from a new-confrontation over last Saturday's American-Soviet "guidelines for an Arab-Israel peace conference in Geneva.

The United States agreed that the U.S.-Soviet declaration "is not a prerequisite for the reconvening of the Geneva conference," but said "it still stands" as the Washington-Moscow version of "the core issues."

An atmosphere of high tension disappeared from American-Israeli relations after six hours of intensive bargaining that ended at 2 a.m. today. President Carter, who participated in the negotiations until midnight, said this afternoon that the talks with Israeli Foreign Minister Moshe Dayan were "very productive." Carter said they will "enhance the possibility of a Genevea peace conference," but "we're obviously going to have a difficult job."

The announced modification of the U.S. position also helped Carter avoid the potential embarrassement of Public opposition from large numbers of senators and House members.

More than 150 House members had signed a letter to Carter sharply criticizing the U.S.-Soviet Middle East statement. The letter will not be sent, House members disclosed, but Carter will meet with a delegation of congressmen Thursday. Similarly, plans for a press conference by about a dozen senators who intended to criticize the administration's policy were dropped after the apparent U.S. shift.

In Israel, the accord on a U.S.-Israeli formula only partly assuaged the dismay generated by Saturday's Soviet-American communique on the Middle East.

"The Americans have taken two steps toward the Arab position and one step back toward Israel," one Israeli official commented.

Although very real and substantive U.S.-Israeli differences remained, the feeling in Jerusalem is that the Carter administration has gone out of its way to paper over and avoid an open confrontation with Israel. The new proposals for clearing away procedural obstacles on the road to Geneva were seen by Israelis as being favorable to Israel.

A senior American official in New York, however, cautioned that "we do not have an agreed basis (between Israel and the Arabs) for going to Geneva."

Secretary of State Cyrus R. Vance now has the task of trying to persuade the Arab nations to go along with the terms that the United States bargained out with Israel. Vance's first attemtp to do so today with Syrian foreign minister Abdel Halim Khaddam reportedly encountered opposition.

A high American official said the United States agreed "we would try it out on the Arabs." Any agreement that Israel accepts quickly, however, is almost automatically guaranteed to be rejected on the Arab side, and vice versa, requiring further bargaining. Sometimes, one side will accept a U.S. mediation plan to precipitate a rejection by the other side. Israel's welcome for the new formula today, only indicated difficult bargaining ahead.

In the bargaining between the Carter administration and Dayan this week. Israel gained assurances that the United States was not trying to pressure it into accepting the American-Soviet guidelines. They include acknowledgement of "the legitimate right of the Palestinian people."

Israel regards that language as an unacceptable concession to the Palestinean Liberation Organization, with which Israel refuses to negotiate. Israel opposed the entire U.S.-Soviet statement as an intrusion by the pro-Arab Soviet Union into the negotiations in order to isolate Israel. The United States and the Soviet Union are cochairman of the conference, which met for only two days in 1978.

The United States, administration officials said, gained from Israel what were described as concessions that now move the negotiations "a little bit further" toward reconvening the Geneva conference with a single, unified Arab delegation "that will include Palestinians."

Israel's Dayan stressed, however, that "we shall not negotiate and have in Geneva the PLO" and "we shall not negotiate for a Palestinian state."

Other Israeli sources said that Israel had granted no significant change in its terms for the conference, and obtained reinforcement for past U.S. pledges to Israel on conditons for new peace talks. Neither U.S. nor Israel officials would disclose the terms of the so-called "working paper."

American sources, however, indicated that Israel had moved from its previous insistence that it would attend only a "ceremonial" opening of the conference with a unified Arab delegation. It apparently had also agreed to more flexibility on how the Arab delegation would then divide into Egyptian, Jordonian and Lebanese delegations and how Palestinians acceptable to Israel could participate in the negotiations.

These issues may seem to be diplomatic hair-splitting, but as a senior White House official pointed out yesterday, "procedural issues to some extent are a mask for substantive issues. This is why procedural issues tend to be so difficult."

Israel's objective, ti to prevent the creation of a separte Palestinian state and also the return to the Arab nations of all the territory Israel occupied in its most successful war in 1967. These are Arab goals supported by the Soviet Union.

In the present maneuvering, Israel has agreed to include Arab mayors and other officials from the occupied West Bank of the Jordon-River in a unified Arab delegation, but not known PLO members.

After the opening ceremony at Geneva, Israel had wanted the delegations to break down to separate national units to exclude any separate Palestinian delegation. The United States is attempting to strike a compromise between Israel and the Arabs on this dispute.

The controversy over the American-Soviet declaration came on top of the running argument over how to convene the conference in Geneva. Even though Israel has rejected the term of that declaration, the United States now evidently hopes it will be credited among the Arab nations for having gone a step in their direction by acknowledging Palestinian "rights" - although these "rights" remain undefined.

In a press conference that began at 2.05 this morning in the United Nation's Plaza Hotel, where Carter and other officials were quartered during their two-day visit, Dayan said, "We do not accept many . . . Provisions" of the U.S.-Soviet declaration of principles, and it was agreed that it will not be "binding" on Israel.

A joint statement issued at the end of talks with Dayan said "the United States and Israel agreed that Security Council Resolutions 242 of 1967) and 338 (of 1973) remain the agreed basis for the resumption of the Geneva peace conference and that all understandings and agreements between them on this subject remain in force." The latter phrase refers primarily to a written accord in 1975 with Israel, at the time it agreed to return a second portion of territory in the Sinai desert to Egypt.

As for the new accord with the United States on a Geneva conference formula, Dayan described it as "a tentative agreement, a provisional agreement, a draft agreement" that he will recommend to this government. It would be effective only if the Arab nations involved also accept it, and Dayan forecasts that ultimately this "probably will be changed too . . ."

Although there have been no leaks as to the specific nature of the new proposals, Israel Radio, reporting from New York, said that they include Israel's continued objection to a Palestinian state - even one lined with Jordan - and that the Americans have said they will not apply economic pressure or pressure in the form of withholding arms.

In Jerusalem, the Foreign Ministry spokesman said today that "until the joint Israel-American statement there was very good reason to fear that the position accepted by the United States in the joint U.S.-Soviet communique would lead to forcing upon Israel new terms of reference for the reconvening of a Geneva conference in contradiction to the agreements of September, 1975."

In other words, the Israel is feared that the Soviet-American communique - by ignoring Resolution 242, which refers to the Palestinian problem only as a refugee problem, and by stressing Palestinian rights - might be used as an attempt to bring the Palestinians and perhaps even the PLO to Geneva over Israel's objections.

The spokesman went on to say, however, that Israel's concerns and objections to the joint Soviet-American communique are still valid concerns." Nor was Israel pleased that Carter again chose to use the term "Palestinian right's in his U.N. speech yesterday. Israel fears that the Arabs interpret "Palestinian rights to mean at the very least an independent Palestinian state and at the worst the dismemberment of Israel."