Israel is quietly scaling down its invasion force in southern Lebanon even before the U.S. peacekeeping force is fully deployed, according to specialists here.

The pullback is yet another sign that Israel is anxious to quiet growing criticism of the controversial invasion at home and abroad and avoid casualties Palestinian commandos could inflict in guerrilla forays.

"The Israelis would pay a great deal to the hell out of south Lebanon - all the way," a Western diplomat remarked, noting that the pace of their withdrawal appeared linked principally to that of U.S. troop deployment.

Intelligence sources here are convinced that no more than 3,500 Israeli troops are still in Lebanon, down from a peak of 5,000-6,000. Palestinian guerrillas claimed that as many as 25,000 troops were involved in the invasion.

Top Israeli commanders had suggested that fewer than 5,000 troops were involved. Any reports of troop strength in the area are subject to strict censorship by Israeli military authorities.

Top Israeli commanders such as Lt. Gen. Mordechai Gur, Israel's chief of staff, privately have expressed disappointment at the pullback's slow pace and indicated formal withdrawal could begin within two weeks.

Even when the pullback is well under way, analysts fear that he lightly armed U.N. force may face potentially exposive problems in the south - principally in dealing with the Palestinians and the Christian militia.

With the scheduled weekend arrival of a 700-man Norwegian contingent, the U.N. Interim Force in Lebanon - already known by its acronym UNIFIL - should be just under half its 4,000-man authorized strength.

With the inclusion of French troops in the U.N. force, two precedents in the Middle East have been set. For the first time, a permanent member of the Security Council has sent troops to the area, and for the first time, a member of NATO has participated in U.S. peacekeeping efforts in the Middle East.

U.N. Secreatry General Kurt Waldheim asked Romania to send troops, but the Warsaw Pact nation refused.

Plans call for the U.N. troops to move south from their present Litani River bridge observation posts and other positions as the Israelis pull back. A U.N. presence would be left in principal villages as the force proceeds toward the Lenanese-Israeli border.

That straightforward southward progression contrasts with earlier, more complicated plans circulated by the Israelis. At one point, Israel had hoped the U.N. force would move into Palestinian-held areas beyond the lines it occupied at the time Israel unilaterally declared a cease-fire.

In a move also backed by the Lebanese government, the Israelis favored moving the U.N. force into Beaufort, Hasbaya and Nabatiyeh - all north of the Litani - and into the Mediterranean port of Tyre.

The United Nations said it lacked the personnel and firepower to fulfill the requests. Moreover, further U.N. Security Council discussions would have been necessary for any such operations, diplomats insisted, and their eventual outcome would have been very much in doubt.

Israel also unsuccessfully floated the idea that Syria would move troops down to the Litani to control the last remaining area of unfettered Palestinian freedom of movement.

Another plan called for the Syrians to move into the Nabatiyeh area located roughly halfway between their present positions around Jezzine to the north and the Litani. Syria, however, was already under fire from the Palestinians and other Arabs for not aiding guerrillas.

Still, diplomats following the southern Lebanese situation are worried that it could turn into a "no-win" operation for the United Nations.

Conventional wisdom suggests that UNIFIL, whose original mandate was for six months, will become a permanent part of the growing U.N. Middle East peacekeeping operations. But the Soviet Union, which refrained from vetoing UNIFIL, is thought unlikely to agree to more than a year's mandate, according to some diplomats.

The Soviets apparently wanted to protect their Syrian friends from getting dragged into a wider war with Israel.

Allowing UNIFIL to continue indefinitely, however, would compromise Moscow's position with the Palestinians by appearing to muzzle their freedom of action in Lebanon.

The Soviets also had sought a provision that would have subjected UNIFIL to removal at Lebanon's request.Since the Security Council resolution establishing the force made no such provision, the Lebanese - or the Israelis - cannot force the troops to leave, as Egypt did with U.N. troops in the Gaza Strip in May 1967.

The United Nations itself is widely believed to be worried about the posibility of a massive return of guerrillas south of the Litani if UNIFIL's mandate should become permanent.

U.N. observers south of the Litani, reportedly have noted isolated groups of Palestinians who either were cut off during the Israeli advance or infiltrated after the cease-fire.

U.N. officials privately are expressing fears that Israel will accuse UNIFIL of tolerating such a guerrilla presence. Such charges could discredit a possible U.N. role in guaranteeing security in the Sinai, for instance around the controversial Rafieh salient.

The southern Lebanese situation is especially delicate because in theory the guerrillas were allowed to station a limited number of men south of the Litani by the Cairo accords worked out under Arab League auspices in 1970.