Carter administration sources said yesterday that the United States has suffered a potentially serious setback in efforts to balance its relations with Israel and the Arab world and fit them into a larger strategy for blocking Soviet moves in the Persian Gulf region.

Although no one would say so publicly, that clearly was the feeling among administration officials trying to assess the effects of President Carter's disavowal of the U.S. vote last Saturday criticizing Israel in the United Nations.

On Capitol Hill, meanwhile, plans for a public hearing on the incident were announced by Sen. Frank Church (D-Idaho). The Senate Foreign Relations Committee, in a session scheduled for Thursday, will call on Secretary of State Cyrus R. Vance and U.N. Ambassador Donald McHenry to give their accounts of the episode, a committee staffer said last night. The panel also will seek written instructions given to the ambassador before the controversial vote.

The administration officials expressed particular concern about the damaging effects of the U.N. incident on the American campaign to win the friendship of Moslem countries as a way of containing the crises in Iran and Afghanistan.

Specifically, they fear that the credibility of this campaign has been tarnished badly by Carter's apparent retreat from an effort to halt an Israeli policy -- building Jewish settlements in occupied Arab territories -- that is opposed vehemently throughout the Moslem world.

That fear appeared to be confirmed by the torrent of criticism that began spilling out of the Middle East yesterday and described Carter in such terms as a "bungler," a "coward" and "a toy in Israel's hand." [Details on Page A14.]

As State Department spokesman Hodding Carter remarked with wry understatement: "We've heard from a number of countries. There has been a reaction that could have been more favorable."

Commenting privately on the Arab reaction, U.S. officials said there is almost nothing the administration can do to counter Arab perceptions that, when it came to a showdown, the United States tilted decisively toward Israel. In terms of Israel, the officials admitted, the net effect of Carter's U-turn on the U.N. vote was to box Washington into a corner where it will be unable to put any real pressure on Israel until after the U.S. presidential elections in November.

Then, echoing U.N. Ambassador McHenry's admission Monday that the United States had wound up with "the worst of all possible worlds," the officials noted that the incident also had strained U.S. relations with Israel to the point where Washington's hopes for an Egyptian-Israel agreement on limited autonomy for the occupied territories could be stalled indefinitely.

In fact, although the administration's official explanation for the turn-about on the U.N. vote laid it to a misunderstanding between Carter and Secretary of State Vance, there is increasingly strong suspicion in diplomatic circles that the president acted, in part, under pressure of an implied Israeli threat to scuttle the autonomy negotiations.

On Monday morning, Israeli Ambassador Ephraim Evron met separately with two influential Carter advisers: Vice President Mondale and Sol M. Linowitz, the president's special mediator for the autonomy talks. Although Evron made no explicit threats, he is known to have warned that the U.N. vote, if left standing, would make Israeli public opinion so suspicious of U.S. intentions that Prime Minister Menachem Begin would be severely restricted in trying to meet the May target date for completion of an autonomy agreement.

Evron's message quickly was relayed to Carter, and it is believed by many sources familiar with the Mideast policy process to have been an important factor in the president's Monday night decision to renounce the U.N. vote.

The negotiations on self-government for the Palestinian inhabitants of the West Bank and Gaza Strip flow from the Egyptian-Israeli peace treaty worked out last year under the U.S.-mediated Camp David accords.

Since that was Carter's greatest foreign policy triumph, he has a strong vested interest in the continued success of the peace process. The success or failure of efforts to meet the May target for an accord on the occupied territories could play a key part in Carter's bid for a second term.

That was underscored yesterday by Sen. Edward M. Kennedy (D-Mass.), Carter's challenger for the Democratic nomination. Kennedy, speaking in New York, charged:

"The Carter administration has managed to jeopardize the security of Israel, damage the peace process in the Middle East, undermine our relations with other friends and make American foreign policy the laughing stock of nations throughout the world."

Some U.S. officials said yesterday that the administration, in seeking to contain the damage to its Mideast policies, now can be expected to put its main stress on trying to get an automomy agreement negotiated by May.

If that can be done, these officials explained, Carter will have impressive new ammunition with which to assert that progress is being made toward Middle East peace despite what the White House calls "the glitch" caused by the U.N. vote.

In addition, the officials added an accord on the occupied territories could be of great help in calming the Arab world's current anger and convincing it that the United States is genuinely concerned about solving the Palestinian problem at the root of the Arab-Israeli conflict.

A swing in emphasis toward the autonomy talks would represent a major victory for Linowitz and others within the administration, who had sought to keep the negotiations between Israel and Egypt sanitized to the maximum degree possible from the larger U.S. effort to win influence among the Moslem countries of the Middle East and Southwest Asia.

For that reason, Linowitz is known to have expressed doubts about the wisdom of backing the U.N. resolution criticizing Israel's settlements policy. In the deliberations leading up to the original decision to vote for the resolution, Linowitz reportedly warned that the timing could have adverse effects on the autonomy talks.

However, concern about the autonomy negotiations came into conflict with the top-priority stress the administration has been putting on its campaign to woo wider Moslem support in the wake of the Soviet Union's Dec. 27 invasion of Afghanistan.

Most key Persian Gulf states like Saudi Arabia had remained chilly toward the U.S. overtures, in part because of their belief that the United States would do nothing to impede what they regard as creeping Israeli annexation of the occupied territories through establishment of Jewish settlements.

Caught between this Arab suspicion and the Begin government's refusal to significantly slow implementation of its settlements policy, the administration decided to vote for the U.N. Security Council resolution as a means of simultaneously registering a protest with the Israelis and demonstrating that it was sympathetic to Arab world concerns.

The intent, U.S. officials have said repeatedly in recent days, was to vote for a resolution criticizing Israel solely on the settlements issue. But, because of the still mysterious "failure in communications" between Vance and Carter, the resolution that finally emerged went further and contained references to the ultrasensitive question of the status of Jerusalem.

That, in turn, provoked a storm of speculation about whether the United States was adopting an overtly pro-Arab policy position, a furious outburst of protest from Israel and its U.S. supporters and, in the end, Carter's about-face statement that the vote had been the result of a mistake.