When a barrage of mortar shells fell a mile inside Thailand and killed five Cambodian refugees last Sunday, the Thai Army was reminded brutally of how easily it might be drawn into open conflict with Vietnamese forces operating in Cambodia.

It is not known who fired the shells. In Bangkok, however, many people assumed that the attack was a warning from gunners -- believed to be Vietnamese -- who last week attacked a major stronghold of the ousted Cambodian government of Khmer Rouge leader Pol Pot in the Phnom Malai hills.

More than 30,000 Cambodians streamed into Thailand after the fighting began last week. Most were civilians, but scattered among them were hundreds, perhaps thousands, of well-fed young men who appeared to be Khmer Rouge soldiers, making another tactical retreat into Thailand.

The day of the mortar attack, a Vietnamese newspaper published harshly worded charges that Thailand was allowing Khmer Rouge soldiers to hide in its territory.

Radio Phnom Penh, controlled by the pro-Vietnamese Heng Samrin government, also escalated attacks on Thai policy.

Thailand responded to the shelling by lodging an official complaint with the United Nations, moving the encamped refugees to a spot three miles deeper inside Thai territory and moving reinforcements opposite Phnom Malai.The Thais did not fire back; as days passed, the crisis appeared to have subsided.

But similar tension is certain to arise again in the months ahead, Indochina watchers believe, as roads dry out and the Vietnamese send tanks and troop carriers against other Khmer Rouge enclaves along the Thai border. a

Thailand is in a predicament since, despite its protestation of strict neutrality in the Cambodian conflict, it has been turned into an important sanctuary by the Khmer Rouge, diplomats say. The Thais do not want or enjoy this role, but they are sheltering large numbers of soldiers who would face almost certain annihilation if they remained in Phnom Malai.

The Thais clearly prefer an independent Khmer Rouge government to one controlled by their old enemy Vietnam.

On the border the official Thai position is based on humanitarianism: people are being allowed in and given food until fighting subsides in Cambodia and they can return home.

In an interview, Defense Minister Prem Tinsulanond said that if soldiers of either side fled into Thailand they would be disarmed and asked to return eventually. On reentering Cambodia their weapons would be given back, Gen. Prem said.

So far only Khmer Rouge soldiers and a few from right-wing Khmer Serei groups operating in Cambodia have entered Thailand this way.

Last spring a column of 80,000 Cambodians, many of them Pol Pot soldiers, entered Thailand ahead of a Vietnamese advance and escaped back to an area where the Vietnamese had not penetrated. Khmer Rouge forces also have been allowed to buy supplies from Thai border settlements.

Thailand continues to recognize the Pol Pot government and has frequently allowed its deputy prime minister, Ieng Sary, to pass through Bangkok on his trips abroad. In the United Nations and other diplomatic forums Thailand repeatedly has called for Vietnam to remove its troops from Cambodia.

Such tilts toward Pol Pot have led to fear that the war could expand into Thai territory. While no one seriously expects an all-out invasion -- in doing so the Vietnamese would over-extend their forces and almost certainly provoke devastating retaliation from China -- many analysts feel that smaller scale fighting close to the border is possible.

Vietnamese units might attempt a lightning thrust into Thailand to encircle Khmer Rouge units resting there or they might choose to attack an isolated Thai unit ot teach the Thais a lesson.

Thailand certainly is not looking for a fight. Indeed as one diplomat said, "The Vietnamese ahve more troops in Cambodia than the Thais have in all of Thailand"

Prem estimates Vietnam's strength in Cambodia at 180,000 to 200,000. Thailand's Army is about 150,000 strong.

Since Vietnamese troops arrived at their border, the Thais have shown remarkable restraint in dealing with border incidents. Artillery shells sometimes fall in Thailand -- several 105mm howitzer shells crashed into a Thai banana grove last week.

Generally, if no one is hurt, the Thais do not respond.

Last spring Heng Samrin troops, believed to be Vietnamese, advanced several hundred yards into Thailand at a border station near Aranyaprathet. The Thai Supreme Command treated it as an accident and several hours later the troops pulled back into Cambodia.

The key question is what risks the Vietnamese are willing to take. Analysts here believe they intend during the next six months of dry weather to wipe out the 20,000 to 25,000 troops still believed to be fighting for the Khmer Rouge.

Vietnam needs a conclusive victory as its economy is in a shambles and it is drawing heavily on Soviet aid to finance to Cambodian venture. Wiping out Pol Pot also would wrap up Vietnamese ambitions of setting up an Indochinese federation of Vietnam, Laos and Cambodia, with Hanoi as the senior partner.

One of the few things holding back Hanoi is the specter of another Chinese incursion into northern Vietnam or possibly Laos, but very few people here are prepared to say the Vietnamese would not go for broke and attack Thailand.