The 12th and possibly last session in the Israeli-Lebanese troop-withdrawal negotiations in southern Lebanon ended today with no sign of progress, leaving Israel to face a series of unattractive choices.

Following today's meeting at the United Nations' southern Lebanon headquarters in the border town of Naqura, Brig. Gen. Amos Gilboa, the head of the Israeli military delegation to the talks, issued a statement threatening to break off the negotiations before their scheduled resumption on Jan. 7.

If Lebanon does not reply positively to Israel's proposals during the scheduled two-week holiday pause in the negotiations, Gilboa said, "the government of Israel will find itself obliged to consider whether there will be any further purpose in the continuation of the talks at Naqura."

However, a senior government official in Jerusalem, noting that Gilboa's statement fell short of an outright ultimatum to the Lebanese, said there remained a "reasonable chance" that the negotiations will resume in January, if only because Israel is unprepared at this moment to take more drastic measures to extricate itself from the southern Lebanon quagmire.

"There is not much to lose in continuing to talk while we prepare for other options," the official said.

At Naqura, the talks have been deadlocked for weeks on the future role of United Nations troops in southern Lebanon.

Israel wants the U.N. force to take control of much of the territory now occupied by the Israeli Army and to serve as a buffer for an Israeli-supported militia that would be stationed in far southern Lebanon. Lebanon insists that its own Army can police southern Lebanon and that the U.N. force should be confined to the area near the international border.

The two sides have not even begun serious discussions of the second major issue in the talks, the role, if any, of the Israeli-backed South Lebanon Army. The Lebanese and Syrians strongly oppose any continuing deployment of this militia in the territory.

For the last several weeks, Israeli officials, including Prime Minister Shimon Peres and other senior policy-makers, have been threatening to abandon the Naqura talks and take "unilateral steps" to secure Israel's northern border. These threats, which have not included an outright ultimatum, have been partly tactical in nature, an attempt to break what the Israelis describe as Lebanon's intransigence in the negotiations.

But the threats also appear to stem from genuine frustration among Israeli officials, who had hoped that the Naqura talks, assisted by U.S. diplomacy in the Middle East, would lead to a negotiated agreement on security guarantees in southern Lebanon and allow them to avoid choosing from among several less palatable options.

Israel's problem in southern Lebanon is, as it has been for more than two years, to find a way to end the costly occupation of the territory while still guaranteeing the safety of Israel's northern communities, which was the announced purpose of the June 1982 invasion.

If those two goals are not reached through negotiations at Naqura, and Israeli officials acknowledge a deepening pessimism that they will be, the most frequently discussed other options comfronting the government here include:

* To stand pat, keeping the Israeli Army along its present line on the Awwali River north of Sidon. This would leave the Army still saddled with the task of policing a large and increasingly hostile area. It also would mark an outright failure for Peres, Defense Minister Yitzhak Rabin and other Labor Party ministers in the national unity government who have pledged to end the occupation of southern Lebanon no matter what.

* To pull back partially in some areas of southern Lebanon, while maintaining the present positions in the eastern sector of the country, where the Israeli Army poses a threat to Syria. The military here has drawn up numerous options for a partial pullback to what Israeli officials describe as a "semi-permanent" new defensive line, where the Israelis would wait until Lebanon and Syria agreed to negotiate a complete withdrawal. Such a course, which many here consider the most likely if the negotiations fail, would amount to half a loaf -- a likely reduction in casualties and in the cost of the occupation, but no end to Israel's direct involvement in Lebanon.

* To withdraw to the international border, in effect sacrificing the goal of security guarantees to get the troops out of Lebanon. This course, however, would run into fierce resistance from the Likud bloc in the national unity government and probably from some Labor Party officials. It was the previous Likud-led government that launched the war, which, so long as it was confined to the goal of securing the northern border, was strongly supported by the Labor Party.

The differences in the government were illustrated this week when Peres, in a published interview, said he personally favored a complete withdrawal.

"I am in favor of the Army pulling back to the international border -- in other words in favor of a full withdrawal and taking the risk," Peres told the independent newspaper Haaretz. "I am not in favor of halfway moves and interim solutions."

The interview prompted an immediate call to Peres from Likud leader Yitzhak Shamir, the government's foreign minister and vice prime minister. According to Shamir, Peres assured him that he remained committed to achieving security guarantees in southern Lebanon before a complete troop withdrawal.

As this exchange illustrated, the three-month-old national unity government has never resolved its internal differences over policy in Lebanon or made any of the hard choices that fall short of its twin, elusive goals of security guarantees and a total troop withdrawal. Instead, this fall the government opted to try once more to achieve those goals through direct negotiations with the Lebanese and indirect contacts with the Syrians.

The Israelis stripped their demands down to what they considered the bare minimum. They no longer insisted -- as they had in the earlier negotiations that led to the now-defunct May 17, 1983, troop withdrawal accord -- that Lebanon establish political and commercial ties with Israel. They also dropped their demand that an Israeli troop withdrawal be accompanied by a near simultaneous Syrian pullout from Lebanon.

Acknowledging that the Syrians held the key to Lebanon's posture in the negotiations, the Israelis also called on the Reagan administration to use its influence in Damascus to bring about a satisfactory agreement.

By all accounts, this two-track approach has led nowhere so far.

Meanwhile, while the Naqura talks have continued fruitlessly, U.S. Assistant Secretary of State Richard W. Murphy has been crisscrossing the region in search of signs of softening by the Syrians. Murphy is scheduled to return to Washington this week without, according to numerous sources, having made any headway.

The lack of progress at Naqura and the failure of Murphy's mission both reflect the strong position Syria enjoys in Lebanon. Israeli officials acknowledge that Syria is the key player on the other side, but they never have been able to answer satisfactorily why the Syrians should cooperate in extricating Israel from its predicament in Lebanon.

Most often, officials here reply that some Israeli units in eastern Lebanon are within artillery range of Damascus and that this surely makes the Syrians exceedingly uncomfortable.

"We assess that this is a big thorn for them," a senior official said today. "They cannot possibly be indifferent to a strong Israeli presence that close to their capital."

The Syrians are undoubtedly not indifferent, but there has been no evidence that they find the situation that has existed for more than two years to be politically or militarily intolerable. From a purely military point of view, the Israeli gunners in eastern Lebanon pose a far less serious threat to Damascus than does Israel's fleet of supersonic fighter-bombers, which could reach the Syrian capital in minutes.

Israel enjoys much less leverage over Lebanon than it did immediately following the 1982 war. Lebanon has lived for more than two years under a de facto partition by Israel. The threat of a partial Israeli pullback does not appear to pose major problems for Beirut.

Some Israeli officials argue that sectarian leaders in Lebanon, fearing an outbreak of violence among the many armed Lebanese factions if the Israelis withdraw, have subverted the Naqura talks in the hope of keeping the Israeli Army, and the relative degree of order it imposes in the territory, in place. Only when these leaders are convinced that Israel is about to pull back unilaterally are they likely to make concessions that would allow a more orderly transfer of security duties to a new force in southern Lebanon, according to this argument.

The increasingly blunt Israeli threats to break off the Naqura talks and take "unilateral steps" have been designed to test this thesis. So far, the threats appear to have been no more convincing than the presence of Israeli Army gunners in eastern Lebanon.

The senior official who was interviewed today suggested that Israel's best option may be to reduce its presence in Lebanon to the minimum, and then settle in for a long waiting game with the Syrians.

"It may take years before they realize we mean business," he said. "But a Lebanon with 3,000 or 4,000 Israeli troops, along with the South Lebanon Army, is not the same as a Lebanon with 30,000 troops, or even 10,000 or 12,000, as we now have there. Israel, economically, politically and militarily, could live with that indefinitely."