Israeli Foreign Minister Moshe Dayan yesterday expressed belief that his talks here showed "a formula will be found" for Palestinian participation in a Middle East conference in Geneva this year.

However, Dayan also underscored the great difference between reconvening the Arab-Israeli conference, which may be only a ceremonial event, and reaching any accord.

Dayan told a news conference that "we will have to distinguish between the start of the negotiations and bargaining, and the ultimate compromise."

There is "a wide gap between us and the State Department" on terms of an Arab-Israeli agreement, Dayan acknowledge. The gap between Israel and its Arab adversaries is even greater.

Today, Egyptian Foreign Minister Ismail Fahmi comes to the White House for the next test of the Carter administration's effort to bring the opposing sides into the direct negotiations.

Dayan's optimism on bridging the defferences over Palestinian representation at a Geneva conference surpassed anything said on the administration's size after 5 1/2 hours of talks on Monday, nearly half of that time with President Carter.

Administration officials said nothing contrary to the Dayan remarks yesterday, only repeating that "it remains our hope to have a Geneva conference this year."

Dayan indicated to reporters that the Palestinian issue could be straddled without altering Israel's outright opposition to any negotiations with the Palestine Liberation Organization, which seeks to establish a seperate Palestinian state.

"The PLO, as such," Dayan repeated several times, would not be a participant in the negotiations which Israel envisions. He said "the United States does not insist on the PlO, as such," attending the conference.

Dayan said:

"I think that eventually, I hope, there will be a way by which the Palestinians, not as a separate group, and not with the idea of having a Palestinian state, but still representing their people, will participate in the negotiations and that the PLO as such will not be represented there."

He noted that Israel long has advocated having some Palestinians from the occupied West Bank of the Jordon River join the Jordonian delegation in peace talks. The Arab nations previously have rejected this approach.

Dayan, however, expanding on what has been said by Israeli Prime Minister Menahem Begin, said Israel would not inquire whether a Palestinian in the delegation is "a sympathizer" with the PLO. "But," said Dayan, "he should not be on the record in any way (as) a PLO representative."

In addition, Dayan questioned, but stopped short of rejecting, an American delegation" at the opening of a Geneva peace conference. The idea behind this U.S. option is to help blur the Palestinian dispute in a Pan-Arab delegation.

Dayan sparred verbally with a reporter on this issue, saying, "What would be the purpose of that unified delegation?" He said, "I can't like or dislike something" when "I don't know what it means."

Any actual peace negotiations, Dayan said, "must be done with each country . . . and not a collective group."

Under the American concept, actual negotiations would be in separate groupings on issues involving Israel's borders with Egypt, Syria, Jordan and Lebanon. An American official yesterday said Dayan indicated he would give further consideration to the "pan Arab delegation" concept.

Dayan told reporters the United States and Israel "will try to find a way by which the Geneva conference can be convened . . . even before the end of the year."

He said, "I do believe that all the parties . . . do want a Geneva peace conference to take place, so I think that ultimately an agreed formula will be found."

"I think," he said, "that we should take at face value" that Egyptian President Anwar Sadat and Jordan's King Hussein "do want to conclude peace talks with us," even though there is disagreement about the nature of the peace.

Israel, he said, of course will remain on guard militarily, to avoid repetition of the Arab's surprise attack in the 1973 "Yom Kippur War." He said he believes, however, that in 1978 Egypt and Jordon "will rely on obtaining their objectives by political means rather than by war," and "the Arab countries won't start a war without Egypt."

The Israeli peace plan he outlined here, Dayan reiterated, will never accept a Palestinain state on the West Bank of the Jordon River, part of Israels "old homeland." Nor will Israel, he repeated, ever return to "the old lines" of Israel's borders that existed before the 1967 war.

But Israel, he said, is "offering the Palestinians on the West Bank to sit down and tell us what they want and how they want to live with us" in that region.

The United States and Israel, Dayan noted, are deeply at odds over Israeli settlements in the West Bank and in other territory occupied by Israel in the 1967 conflict. There are about 80 settlements now. But Israel, he said, "understands that the settlements cannot determine that ultimate border."

In addition, Dayan pointed out, the United States wants "almost a complete withdrawal" of Israel from the occupied territories. The Arabs demand total withdrawal.

Nevertheless, Dayan said, "the differences of position "are such that I believe we can live with" them. He spent nearly an hour privately with President Carter and Vice President Walter F. Mondale on Monday, and Carter then joined in another hour and 35 minutes of Dayan's talks with Secretary of State Cyrus R. Vance and other officials.

Asked yesterday if he felt under any "American pressure," Dayan said the United States "will try to convince us" but "I don't think they will try to pressure us."

However, a contrary view came from Sen. Richard Stone (D-Fla.), who heads the Senate Foreign Relations Subcommittee on the Near East. After a committee luncheon for Dayan, Stone said "it is to be deeply regretted" that the State Department had chosen "to lean toward a PLO formula" in dealing with Israel, on the eve of Dayan's visit to the United States.

The PLO, said Stone, is an "inbred terrorist organization" and "bringing in the PLO" into negotiations "is putting the Russians between Jordan and Israel, threatening the vital interests of both countries."

On another subject, which has hung mysteriously over the prelude to Dayan's visit here, his travels last weekend, Dayan told reporters after the Senate luncheon that "I did not see King Hussein over the weekend."

Dayan hinted at his earlier news conference that he had met with a senior Arab representative in Europe, causing him to double back to Israel on his way to the United States. Dayan said he would like to say "I did meet or I did'nt meet with a certain personality, but you have to realize that it concerns other people too and they are not eager to discuss it."