The Palestine Liberation Organization has said that, unless the United States recognizes it or acknowledges Palestinians' right to self-determination, it will not consider naming non-PLO members to an Arab delegation for peace talks with Israel, U.S. and Arab sources said yesterday.

Assistant Secretary of State Richard W. Murphy, just returned from a 2 1/2-week tour of the Middle East, tried to meet with such a delegation that could have served as a model for future direct Arab-Israeli talks, the sources said.

The State Department has refused to comment on what, if anything, Murphy accomplished during exploratory talks with Arab leaders on ways to revitalize direct Arab-Israeli talks but has insisted that "neither dramatic breakthroughs nor setbacks" occurred.

Other Arab and U.S. officials said, however, that Murphy made no progress on devising a formula for a joint Jordanian-Palestinian delegation that might participate in talks with Israel. Many believe that the PLO position is hardening against the idea of non-PLO members participating in such a delegation without something in return from the United States.

"We are back to square zero," one Arab diplomat said.

According to initial reports about a PLO meeting in Baghdad April 16-17, PLO chief Yasser Arafat won a narrow vote, either 6 to 5 or 5 to 4, in the executive committee in favor of delegating non-PLO members to a Jordanian-Palestinian delegation to meet with Murphy. But he subsequently decided against it for fear of splitting the PLO leadership, sources said.

Both the PLO executive committee and the Central Committee of Fatah, the Arafat-led main faction in the umbrella Palestinian organization, met in the Iraqi capital while Murphy was touring the region.

"Arafat had a slight majority, and he decided against it to avoid a split," a U.S. official said of the Baghdad vote. "He's already had his split and didn't want to lose more," the official added, referring to the defection of pro-Syrian and other factions from the PLO over the last three years.

Since the Baghdad session, there have been multiple signs of a hardening in the PLO line toward accepting any proposed Jordanian-Palestinian delegation without direct PLO representation, U.S. and Arab analysts agree.

U.S. officials regard the attempted seaborne attack April 20 by Fatah commandos on Tel Aviv, which an Israeli patrol boat aborted, as particularly significant and indicative of the Arafat faction's intention, since it apparently was planned to coincide with Murphy's mission.

"It would have been the largest seaborne operation Fatah ever pulled off," a U.S. official said.

Another close Arafat aide, Salah Khalaf, has said the Baghdad meeting rejected a Murphy proposal that the PLO recognize Israel as a precondition for U.S. acceptance of the PLO in a joint Jordanian-Palestinian delegation and another asking it to choose non-PLO members to participate instead.

U.S. and Arab sources said Arafat has conveyed several "counterproposals" to the United States that Washington make some concession to the PLO first: either recognizing the PLO as legitimate representative of the Palestinian people or recognizing the Palestinian right to self-determination.

The United States has a longstanding policy of nonrecognition of the PLO unil the PLO accepts the state of Israel.

Palestinian self-determination is difficult for Washington to accept because it has become a code word in the Arab world for an independent Palestinian state. President Reagan, in his first-term Mideast peace initiative, said the United States would not support this idea.