As the U.N. Security Council prepares to renew its peace-keeping mandate in southern Lebanon next month, the multinational forces there say they are frustrated more than at any time during the past seven years by a gradual erosion of their assigned territory and a diminution of their responsibilities.

In real terms, the U.N. Interim Force in Lebanon (UNIFIL) is worse off than it was in June 1978. At that time, the Israeli Army withdrew from southern Lebanon and disregarded the U.N. force by turning over an arbitrarily drawn border strip to an Israeli-backed Christian Lebanese force instead of to the U.N. peace-keepers, as called for by Security Council Resolution 425, U.N. officials complain.

Now, senior U.N. officers at the headquarters here say, the area the peace-keeping forces can functionally control without interference by the Israeli Army and its Lebanese militia has shrunk even further. It is 10 to 15 percent smaller than the area it controlled from 1978 until June 6, 1982, when Israeli forces invaded Lebanon for the second time in four years.

On the maps, the U.N. zone is still the same size, but encroachment by the Israeli Army and its Lebanese allies is steadily nibbling at UNIFIL's area of control, the U.N. officials say.

According to U.N. officials, there are more Israeli troops in southern Lebanon now than at any time since Israel ostensibly completed its withdrawal in June.

Pointing to a map dotted with pins denoting positions in the U.N. zone held by troops of the Israeli Army and its Lebanese militia ally, the South Lebanon Army, UNIFIL's deputy chief of operations, Col. Eystein Singstad, said the Israelis "say they have no responsibility for what happens in the [U.N.] area, but I think now they will more and more take responsibility. They have so many units in the area that they can't deny being there."

He added, "The chances of fulfilling the mandate are very poor when one party doesn't accept the presence of UNIFIL in the area."

Israeli officials refuse to discuss the number of troops they have in southern Lebanon, saying only that they maintain liaison units at the South Lebanon Army's headquarters at Marjuyun, man some observation posts inside Lebanon and occasionally make "search-and-arrest" forays to counter terrorist infiltrations into Israel or to prevent attacks on the Christian militia.

But Singstad said in an interview that the Israeli Army maintains a fully mechanized battalion of 400 men near Hasbayya, north of the Norwegian U.N. battalion's area of control; a mechanized infantry company of 100 men north of the Crusader-era Beaufort Castle in a region where Finnish peace-keepers are deployed, and a company of 100 troops in the Finnish battalion's zone that is equipped with Merkava heavy tanks with sophisticated night-vision apparatus.

Singstad said the total of 600 Israeli troops inside or north of the 300-square-mile U.N. area of operations is greater than the number of Israeli troops in the 7-to-10-mile-wide security zone the Israelis set up just north of their border when they withdrew in June. The security zone is controlled by the South Lebanon Army, which is Israeli-equipped and headed by Antoine Lahad, a former brigadier general in the Lebanese Army.

Israeli Army officials say Lahad has 2,000 militiamen in the security zone, but U.N. officials say it is closer to 1,000. The number of Israeli troops in the zone fluctuates sharply, depending on Lahad's need for tactical support, but is believed to be several hundred men most of the time, U.N. officials here said.

Singstad, of Norway, said the encroachment in the U.N. zone is especially frustrating because of an understanding the Israelis have with Lahad: South Lebanon Army troops are not supposed to enter the U.N. zone unless they are accompanied by the Israeli Army, whose withdrawal the U.N. force is supposed to be confirming under the Security Council mandate.

Originally charged with responsibility for preventing infiltration from Lebanon to Israel of Palestinians and radical Shiite forces, UNIFIL now has been reduced to a role of preventing infiltration from one part of Lebanon to another, Singstad conceded. He said the force is further away than ever from fulfilling the dictate of the mandate to assist the Lebanese government in restoring its authority in the area.

"I can't say, unfortunately, that we can accomplish what we came here to do unless we are allowed to deploy to the border," Singstad said. U.N. Secretary General Javier Perez de Cuellar, according to U.N. sources, recently said essentially the same thing to delegates to the Security Council in preparation for the Oct. 19 vote on extending the mandate of the force.

Singstad said he was confident that the dominant Shiite Moslem militia, Amal, would "work closely" with the 5,800 U.N. troops in southern Lebanon to maintain peace and prevent infiltration into Israel by Palestinian guerrillas or members of more radical Shiite factions, such as Hezbollah.

Senior officials of the Israeli Army northern command say that Amal's intentions to disarm Lebanese gunmen in southern Lebanon would crumble under pressure from radical groups like Hezbollah and the Syrian-supported Syrian National Socialist Party, which are bent on attacking Israeli settlements in Galilee.

The Israelis point out that radical militias have detonated 11 car bombs along the edge of the security zone since Israel withdrew most of its troops in June and that Katyusha rockets are fired almost nightly toward Israel from the Amal-controlled areas north of the U.N. zone.

UNIFIL field commanders said in interviews that if they were allowed to deploy to the border, they could gain Amal's cooperation and maintain security in southern Lebanon.

"Amal just wants the Israelis out of Lebanon," said Maj. Ger Pastoor, commander of a Dutch U.N. company in the village of Majdal Zoun. "They don't seem interested in attacking in Israel. Nobody from Amal in this area wants shooting from their area, because they know they'll just get it back from Yarin." At Yarin, the South Lebanon Army has a major artillery battery, with 155 mm howitzers.

Lt. Gerald Schienbroek, commander of a Dutch U.N. position at Ras Biyada, on the coast near the Israeli border, said there are nightly attacks on South Lebanon Army positions in his area by Amal units north of the U.N. zone, but he said it is usually ineffective firing.