Israeli invasion of Southern Lebanon has put new strains on the uneasy relationship between Syria and the Palestinian guerrillas and revealed internal pressures on the government of President Hafez Assad.

Embarrassed by its inability to give more than token military assistance to the Palestinians, Syria is also angry at the Palestinians for having put this country in such as awkward position.

"The Palestinians have to make trouble or nobody would pay any attention to them," an official on Assad's staff observed. "But there is a proper time for it."

Officials here are even more reticent than usual in discussing Syria's role in Lebanon, but they admit that Assad is walking a narrow and treacherous path between his needs to restrain the Palestinians and to continue presenting himself as their patron.

According to Syrian, Palestinian and foreign diplomatic sources here, Assad was being urged by influential political and religious figures to join the Palestinians in their fight against Israel.

At the same time, however, Syrian interests lie in avoiding war with Israel, reducing tensions in southern Lebanon so Israel will withdraw and bringing enough stability to Lebanon so Syria can pull out the 20,000 troops it has stationed there since the civil war ended in 1976. None of those interests would have been served by intervention on the Palestinian side during the Israeli invasion.

What Assad has done is put diplomatic and logistical pressure on the Palestinians to persuade them to stop shooting in the south while continuing to proclaim Syrian support for their cause and avoiding a direct confrontation between his armed force and theirs. In this way he has sought to avoid any appearance of collaborating with Israel and the Lebanese Christians in a campaign to break the back of the Palestinian resistance.

It has not fooled the Palestinians, and in the words of one Western diplomat here, "It won't fool anybody." But Assad apparently had few alternatives.

Syria's role is crucial to the unfolding drama in Lebanon because its troops represent the chief police authority in Lebanon and because it has supplied weapons and equipment as well as political support, to the Palestinian guerrillas. Last year the Syrians forced the Palestinians to accept an agreement that would limit their activities in southern Lebanon in the hope of fending off exactly the kind of invasion Israel has now carried out.

Syria restored order in most of Lebanon after the civil war but never moved its army into the south because of Israeli objections. Even if it had, it could not stop the kind of terrorist raids that prompted Israel to move. But it was assumed in Lebanon that Syria could oblige the Palestinians to accept a cease-fire, as it in fact did, to expedite the deployment of U.N. troops.

Assad signaled the approach he was going to take to the southern Lebanon crisis in a recent interview with Newsweek. He said Syria would allow weapons and equipment to flow across the border to the Palestinians fighting in Lebanon but he also insisted that "Israel's plan will not draw us into positions of adventurism."

This showed up in practice with the announcement that the Arab peacekeeping force in Lebanon was cutting off the flow of weapons and supplies to the Palestinian in Lebanon in an effort to bring the situation under control. That force is dominated by Syria and virtually dictates policy in Lebanon, but the Syrians insisted the next day that the decision was made by Lebanese President Elias Sarkis, not by them.

Their policy, they said, was to continue helping the Palestinians.

Observers here say that Assad has been under pressure to encourage and support the Palestinians, rather than curb them, even if meant taking on the Israelis.

Some officials of the civilian wing of the ruling Baath Socialist Party were reportedly arguing that Syrian forbearance and caution had resulted only in the occupation of more Arab land by Israel, and that it had been time for a dramatic riposte to provoke a Middle East cataclysm.

"Ever since the 1973 war we have been patient. Look at the situation of the Golan Heights, not one shot did we fire at them. And we said we were ready for peace. But now we see their response and it's time for us to act. We have to choose," said a prominent civilian official who espoused this view.

But Syria's military commanders, and Assad himself, are reliably reported to understand that this country would be devasted in a war with Israel, especially since Egypt might sit it out rather than help.

There apparently was also some religious pressure on Assad if not to intervene directly on the Palestinians' side, at least to give them more assistance.

Assad is a member of the minority Alawite sect of Islam, which represents only about 10 percent of the population and is scorned by the majority Sunni Moslems.

A dozen or more prominent Alawites have been assassinated in the past year, and Assad reportedly fears that factional strife might jeopardize the unity of the nation. Under the circumstances he could not allow the Sunnis to think he was supporting the Jews of Israel and the Christians of Lebanon against the Moslem Palestinians.

Furthermore, Syria continues to aspire to political leadership of the Arab world, an aspiration that is at least partly responsible for this country's long feuds with Egypt and Iraq, rival claimants.

In playing that game, it is important to be seen as the champion of the Palestinian cause. The Syrians were jolted, according to sources here, when Palestinians in the Yarmouk refugee camp outside Damascus rioted in protest against Syria's alleged failure to do enough for the guerrilla fighters who were standing up to the Israelis in Lebanon. Three persons died in that incident.

Despite all those pressures, Assad bowed to two superior realities in deciding to stay out of the southern Lebanon action, observers here say. One is the clear military superiority of Israel. The other is Syria's military, economic and political need to extricate itself from involvement in Lebanon, not to plunge in deeper.