Vietnamese forces occupying neighboring Cambodia launched a major counterattack today against anticommunist guerrillas trying to expand their strongholds in the western part of the country. Thailand threatened to retaliate if Hanoi's troops pushed their offensive across the border.

Thai forces stationed near the Cambodian border were put on alert and reconnaissance planes were sent to drop flares marking the frontier as a warning to the Vietnamese, military officials said. Thai troops also were reported to have fired smoke shells to warn the Vietnamese that a number of artillery shells had landed in Thailand.

Vietnamese troops backed by artillery and tanks launched the assault against guerrillas of the Khmer People's National Liberation Front in the village of Phum Yeang Dangkum a few miles east of the front's border camp of Nong Chan. Thai officials said the Vietnamese today recaptured the village, which the front's fighters had seized after attacking a Vietnamese garrison there on Dec. 26.

The escalated fighting in the border area in recent days appeared to signal the start of Vietnam's expected dry-season offensive against resistance groups opposing Hanoi's occupation of Cambodia.

The strongest guerrilla group, the communist Khmer Rouge, also has been reporting increased fighting lately. Ousted by the Vietnamese in January 1979 after nearly four brutal years in power, the Khmer Rouge now participates in an uneasy coalition with the anticommunist Khmer People's National Liberation Front led by Son Sann and a group loyal to former Cambodian head of state Prince Norodom Sihanouk.

The Vietnamese assault began early today with the shelling of the front's Nong Chan camp, inhabited by about 40,000 refugees. The camp was reported to have been hit several times, but casualty figures were not immediately available.

According to the chief of Thailand's National Security Council, Squadron Leader Prasong Soonsiri, the government already has ordered the Thai Army to retaliate if Vietnamese forces cross the border. Prasong said Thailand would not allow any Vietnamese or Khmer Rouge forces into its territory.

"We will push them back," he said in an interview today. "If we have to fight, we will fight."

Prasong said there were many indications that the Vietnamese are about to launch a "big offensive" near the border. But he said the terrain was still too wet for Vietnamese tanks and armor to maneuver easily and that the major push might come later in the November-to-May dry season.

Although Hanoi has denied preparing for an offensive, western and Thai military analysts assert that the Vietnamese have been bringing fresh troops and new equipment including armor and artillery up to positions near the Thai-Cambodian border in recent weeks.

According to Prasong, Vietnam now has as many as 40,000 troops near the border along with new long-range 130mm Soviet-supplied artillery and some T54 tanks. He said the Vietnamese also have been rehabilitating an old airfield at The Beng and building a number of helicopter pads in western Cambodia.

A western military attache said the Vietnamese have been replacing many of their captured U.S. 105mm artillery with the Soviet 130mm pieces, which have a range of more than 16 miles. "Even the Thais have nothing that shoots that far," he said.

Another western diplomat said the Vietnamese appeared better organized this year and probably would be able to carry out operations on a larger scale than last dry season, when their offensive did not go very well.

"They ought to do a better job this time around," he said. "All indications are that they have the men and materiel to do it."

Western and Thai military analysts also expect the Vietnamese to concentrate more of their efforts against the noncommunist resistance groups this year, although they represent less of a threat than the estimated 30,000 hardened guerrillas of the Khmer Rouge. The groups of Son Sann and Sihanouk combined field about 14,000 to 16,000 armed fighters, the analysts said.

"I think the front rightfully has the Vietnamese worried," said a western military attache.

He argued that the noncommunist groups have the greater potential to develop self-sustaining insurgencies. The Khmer Rouge "no longer have the capacity to rally people, but Sihanouk and the front do," he said.

Son Sann's front also has markedly stepped up its activities lately, carrying out raids such as the one that briefly captured Phum Yeang Dangkum.

In its latest communique, the front claimed successful ambushes in the region north of Highway 5 between Poipet and Sisophon.

It said that on Dec. 31, its forces attacked a Vietnamese position at Beng Ampil, 3 miles south of Ban Sangas, burning three barracks and causing "many" enemy casualties. On Jan. 1, the communique added, the front's guerrillas clashed for several hours with Vietnamese troops nine miles north of Sisophon.

There was no independent confirmation of the claims, but Thai military officials have reported scores of casualties on both sides in recent days as fighting has flared between the Vietnamese and the different Cambodian groups.