Israeli troops captured commanding heights overlooking this Mediterranean port yesterday, the fifth day of their punitive operation against Palestinian guerillas in southern Lebanon.

With complete control of the ridgeline villages fewer than three miles away, capturing Tyre looked well within the Israelis' grasp if they are willing to pay the price in casualties.

Some specialists suggested that Israel would be satisfied with isolating the port, which Israeli leaders have marked as the main entry for Soviet weapons destined for the Palestinians.

Palestinian officials insisted that the Israeli goal was the total occupation of the area south of the Litani River, a tenth of Lebanon's territory.

Even more worrying in terms of a possible wider war involving Arab states were the first Israeli air sorties over the northern Lebanese city of Tripoli. Syria announced Saturday that it was instituting earial reconnaisance patrols over all but the southernmost strip of Lebanon, and the risk of a serious incident with Israel planes has prompted serious concern.

Behind Israel's decision to expand what was originally advertised as a limited security zone operation, analysts believed, was a desire to hold as much of southern Lebanon as possible as a bargaining counter in whatever negotiations may lie ahead.

Israeli officials had made no secret of their hopes of stalling yesterday's U.N. Security Council vote, which called for a cease-fire and Israeli withdrawal, until Prime Minister Menachem Begin sees President Carter in Washington.

The fifth day of fighting was the most violent yet.

Backed by air strikes, heavy artillery and naval gunfire, Israel armor and infantry fought their way up two mountain valleys from the east. Down on the coastal plain Israeli troops kept up pressure on the Palestinians from the south.

For the first time in this conflict, Israeli warplanes dropped cluster bomb units, which scatter shrapnel across a football, field-sized area.

The target was the Rashidiyeh Palestinian refugee camp. Hours after the raid, an Israeli helicopter flewover the camp south of Tyre no more than 20 feet off the ground and was not fired upon.

Hit with conventional high explosive bombs and heavy artillery - possibly U.S. - supplied 175 mm guns - was the Bourgh chemali refugee camp due east of the port area.

The camp may have been singled out because shoulder-held Soviet-made Strela antiaircraft missiles were seen being fired from there - unsuccessfully - at Israeli warplanes.

Artillery, naval gunfire and air strikes were also directed against Palestinian positions of the slopes of the last range of hills before the sea.

For the first time in this sector, machine gun and assault rifle fire could be clearly heard - a sign of clos-in combat.

Even before Lt. Gen. Mordechai Gur, the Israeli chief of staff, announced that his troops had penetrated as far as 13 miles inside Lebanon, Palestinian commandos were pulling back north of the Litani River.

But no confirmation was immediately available of Gur's assertion that Israeli troops had cut the key Zaqiye Bridge across the Litani, southwest of Nabatiyeh, which was part of the main palestinian supply route for troops in the south.

By early morning trailer trucks were carrying loads of ammunition out of the area. A steady stream of field guns, antiaircraft weapons, heavy mortars and other crew-served weapons were also evacuated.

Palestinian troops - and their leftist Lebanese allies - drove out on a variety of vehicles or walked along the roads and through the coastal orange groves.

The massive Israeli attack also emptied Tyre and the surrounding vilages of almost all those few inhabitants who had not fled north since the Israelis invaded just before midnight Tuesday.

The seriousness of the situation was illustrated by the presence of bedouin nomads - normally unmoved by the cyclical Middle East violence - who moved everything including camels, sheep and boats north on the main coastal road.

Thousands of refugees drove north crammed in taxis, buses and trucks and perched on tractors or hanging on for dear life in open car trunks.

In Damour, only 12 miles from Beirut, Syrian peacekeeping troops caused an hour-long traffic jam by trying to turn back refugees who had brought bedding with them.

It was apparently their rough and ready way of discouraging squatters from storming empty Beirut apartments, following the first wave of refugees now thought to total as many as 200,000 persons.

At one point on the road between Tyre and Sidon the traffic was so dense that Palestinian guerillas fired into the air to clear a passage for their car carrying a wounded man to the hospital.

On the diplomatic front, Palestinian exasperation at their Arab allies surfaced in a statement by the Marxist Popular Front for the Liberation of Palestine.

"Five days have passed since the Israeli aggression and the Arabs are still acting as spectators" it said.

In Damascus, foreign and defense ministers from the Palestinians' allies in the anti-Egyptian front set up last December were meeting in an effort to respond to such criticism.