Israel's Cabinet formally adopted a policy in southern Lebanon today that calls for U.S. mediation to achieve security guarantees and a troop withdrawal and rules out, for the next several months at least, a unilateral Israeli pullout.

The decision, announced in a Cabinet communique, had been expected and was based largely on a four-point negotiating position made public two weeks ago by Prime Minister Shimon Peres. Even by optimistic assessments, negotiating an agreement based on the four Israeli demands and withdrawing the troops could take a year or more.

In Washington, the State Department said Sunday that U.S. participation is unlikely at the moment. "We have addressed the question in the past in that we're not playing a mediating role," a spokesman said. "There are a lot of things that have to be settled before we can come close to a negotiating or mediating role."

The Israeli conditions for a troop withdrawal call for Syria to guarantee not to move into areas vacated by the Israeli Army and to prevent guerrillas from infiltrating south toward Israel from Syrian-controlled territory. They also call for the continued deployment of the Israeli-backed South Lebanon Army in far southern Lebanon and the redeployment of United Nations troops to a zone north of the South Lebanon Army positions.

Israeli officials said no time limit was set for achieving a withdrawal accord based on these conditions, although they acknowledged that there are "human limits" to how long they would persist in unproductive talks.

In adopting this stance, Israel apparently hopes to persuade Syria that the only hope for an Israeli troop withdrawal is through indirect talks mediated by the United States. This is likely to be the Israeli position during the coming months in Lebanon, where the Israeli Army is digging in for a third winter. But if the effort fails, Israel again will be faced with a decision on whether to pull back unilaterally from some parts of southern Lebanon.

Under the Israeli policy adopted today, the United States will be asked to intercede with Syria for an unsigned accord while Israel seeks direct military talks with the Lebanese government on security in southern Lebanon.

Both parts of this two-pronged negotiating effort appear to be in trouble, however. The Reagan administration has said it is willing to help arrange a troop withdrawal agreement, but only if all of the parties involved request U.S. help and there is a reasonable chance of success. So far, according to officials in Washington, these conditions for a U.S. role have not been met.

However, aides to Peres said that during a visit here last week Defense Secretary Caspar W. Weinberger indicated that the administration could take a more active role in attempting to break the stalemate over Lebanon after the November presidential election.

Meanwhile, Israel's first efforts to arrange direct military talks with the Lebanese have become bogged down in procedural disputes. Lebanon insists that such talks take place under the aegis of the Israeli-Lebanese Armistice Commission, the body that oversaw the 1949 cease-fire between the two countries.

Israel has rejected this because it sees it as a symbolic step back to the country's first struggling years as a new state in the Middle East. The Israelis have said they are willing to meet with Lebanese military officials under U.N. auspices but insist that the talks be direct.

These procedural squabbles are only the first barrier to achieving the security arrangements Israel is seeking in southern Lebanon.

In an interview published Friday, Defense Minister Yitzhak Rabin said negotiating the security arrangements could take months. Implementing an Israeli withdrawal from Lebanon would take six to nine months more.

Rabin said that if withdrawal negotiations are successful, Israel plans a two-stage pullback, first leaving its current positions along the Awwali River to a new line further south, probably near the Zahrani River. United Nations troops, which currently are deployed only as far north as the Litani River, would then be moved north into the vacated Israeli positions. Israel would stay along this new line for "several months," Rabin said, "to test whether the security arrangements are being implemented."

In the final stage of the withdrawal, Rabin said, Israeli troops would pull back to the border while units of the South Lebanon Army would take control of the territory south of the U.N. zone.

Virtually every aspect of this plan requires the acquiescence of Syria. The Syrians also could thwart the plan by pressuring the Lebanese government not to enter military talks with Israel.