Heavy Vietnamese shelling halted a new counterattack today by Cambodian guerrillas determined to recapture their largest camp on Cambodia's western border with Thailand, according to Thai military and resistance sources.

As fighting at the Rithisen camp opposite the Thai border village of Nong Samet continued for a 10th day, the guerrillas of the anticommunist Khmer People's National Liberation Front appeared to be locked in a costly stalemate with more heavily armed Vietnamese forces backed by armor and artillery.

However, Khmer front officials indicated that the guerrillas were determined to battle on for the camp in the hope that Vietnamese supplies would run out or be cut off, forcing a withdrawal.

But with no indication yet of such a pullout, the guerrillas evidently have been taking a pounding from Vietnamese artillery and mortar fire and have failed to gain ground.

Red Cross officials reported that 54 Cambodians were wounded in fighting at the camp last night and this morning, most by shelling. Of that number, the officials said, 19 had to be evacuated to a hospital at the Khao-i-Dang refugee camp inside Thailand, while the rest were treated close to the border at an evacuation site where about 62,000 civilian inhabitants of the Rithisen camp now are living under makeshift shelters.

The Red Cross had no figures for those killed in the fighting.

Khmer front leaders put today's casualty toll at five guerrillas killed and about 20 wounded. There were no estimates of Vietnamese casualties.

With more than 50 guerrillas reported killed and about 250 wounded, Khmer front losses in the current battle have been the heaviest of any operation since the group was formed five years ago to battle the Vietnamese invasion of Cambodia. But the front's decision to keep throwing forces into the battle may be intended to preoccupy the Vietnamese and divert them from an attack on the more strategically important Ampil base 18 miles to the north, where the front has its military headquarters, diplomatic sources said.

Nevertheless, there have been unconfirmed reports that the Vietnamese are beefing up their forces near Cambodia's northern border with Thailand and may hit other bases of the Khmer front, the largest noncommunist group battling the Vietnamese.

The front has at least 12,000 and possibly as many as 18,000 armed fighters (the latter figure is claimed by front leaders), while a smaller noncommunist faction led by Prince Norodom Sihanouk fields about 5,000 guerrillas. The strongest resistance group, the communist Khmer Rouge, has 30,000 to 40,000 hardened guerrillas and is considered the main military threat to the 160,000 to 180,000 Vietnamese troops occupying the country.

While the Ampil camp was quiet today, a small skirmish between Khmer front guerrillas and Vietnamese troops was reported at a nearby lake southeast of the base.

In addition, according to Thai military sources, clashes took place three miles south of Rithisen near the front's Nong Chan base. Vietnamese troops overran the base following an attack Nov. 18 to begin their current dry-season offensive.

The front said it reoccupied Nong Chan last month after the Vietnamese withdrew, but the settlement's 20,000 civilian inhabitants remain at an evacuation site on Thai territory.

In a separate development, Vietnam today commuted to life imprisonment the death sentences of two men, including a French Vietnamese national who was convicted along with 19 others last month in Ho Chi Minh City of plotting to overthrow the Vietnamese government with help from China, Thailand and the United States.

The two were among five alleged rebels sentenced to death by firing squad after a public show trial. The fate of the other three was not immediately clear.

The official Vietnam News Agency reported tonight that appeals for clemency by Mai Van Hanh, a dual French Vietnamese national in his late fifties, and Huynh Vinh Sanh, had been granted. They and the three others sentenced to death Dec. 18 became the subjects of international appeals to Hanoi, particulary from France.

The other 16 defendants received jail terms ranging from several years to life imprisonment. China, Thailand and the United States all denied Hanoi's charges of complicity in the alleged plot.

In a separate trial Dec. 27, three other Vietnamese received death sentences on similar charges.