U.S. special envoy Philip C. Habib returns to Jerusalem Wednesday and to an atmosphere of mutual suspicion between Israel and the United States that has been generated by the stalemate over negotiations for a troop withdrawal from Lebanon.

The suspicions were plainly evident from a flurry of charges and countercharges by anonymous officials on both sides here over the last several days. But there have also been signs that the deadlock has become troublesome to the government of Prime Minister Manachem Begin.

A source close to Begin said today that the Israeli government "wants things to calm down so we can talk business." On one of the key issues blocking the start of troop withdrawal negotiations -- Israel's insistence that some of the talks take place in Jerusalem -- the official said "there are possibilities to solve this," suggesting a softening of the Israeli attitude on the issue.

Over the weekend, U.S. officials suggested that the Jerusalem question be avoided at the outset by starting with indirect "shuttle negotiations" in which Habib and fellow American diplomat Morris Draper would travel between Beirut and Jerusalem. A reported alternative plan is for Israeli and Lebanese negotiators to meet directly on neutral ground somewhere in Europe.

Israeli sources said they expect the Begin government to find the European alternative more attractive because it would still involve face-to-face meetings with the Lebanese. That impression was reinforced when Deputy Foreign Minister Yehuda Ben-Meir told reporters today that Israel will "listen closely" to any new ideas suggested by Habib and that Israel's prime objective is to reach an agreement through "direct negotiations" with the Lebanese.

The open U.S.-Israeli differences that have preceded Habib's arrival here stem largely from what each country wants the negotiations to lead to besides the withdrawal of Israeli, Syrian and Palestinian troops from Lebanon.

To the Reagan administration, the continuing deadlock in Lebanon is a major roadblock to its larger goal of pressing ahead with President Reagan's Middle East peace initiative. So long as Israeli troops remain deep inside Lebanon, there is little chance that other Arab leaders will be willing to join negotiations on the future of the occupied West Bank and Gaza Strip.

The Begin government rejected the Reagan plan out of hand and has no desire to give it any momentum. At the same time, it has greatly expanded its ambitions in Lebanon, transforming them from the initial announced military goal of freeing Israel's northern border from the threat of Palestinian attack to the broad political-diplomatic objective of achieving "normal relations," if not a formal peace treaty, with a second Arab country after Egypt.

Israeli officials, viewing their military presence in Lebanon as leverage to achieve their larger goal, speak of there being "almost ideal conditions" for peace between Israel and Lebanon if only the United States would exert more pressure on Beirut.

The American perspective was outlined to reporters here last week under ground rules that the information be attributed only to "informed sources." The thrust of the message was that the haggling over procedural issues on how to begin the troop withdrawal negotiations has led to frustration among Reagan administration officials and dark suspicions about Israel's true intentions in Lebanon.

How are the Israelis' objectives in Lebanon to be viewed, reporters were asked, when they constantly proclaim their eagerness to leave Lebanon yet throw up an endless series of roadblocks to beginning the negotiations that could lead to that end?

The sources suggested that the Israelis may be deliberately stalling, the better to fend off the Reagan plan, and might be perfectly content to remain in Lebanon indefinitely. The stalemate could well produce a U.S.-Israeli "confrontation," the sources said, suggesting that political figures in both governments might welcome just such an open clash as a rallying point for their own domestic constituencies.

Israeli officials replied that this was all "rubbish," charging that the United States, by pressuring Israel on the troop withdrawal question, hoped to persuade Jordan's King Hussein to join the broader Middle East peace negotiations envisioned by the Reagan peace initiative.

Defense Minister Ariel Sharon reiterated that charge today. Speaking to reporters in the West Bank, Sharon blamed the stalemate on U.S. attempts to "connect" the Lebanon negotiations with the Reagan plan. He said any settlement must be a "direct arrangement" between Israel and Lebanon and include "a signed treaty between the two governments."