Vietnam has wound down its biggest offensive since invading Cambodia nearly 3 1/2 years ago, and Cambodian guerrillas have retaken some key positions recently abandoned by Hanoi's troops, according to Western diplomats and Thai military sources.

The Vietnamese captured the positions in a major dry-season thrust into areas of western and northern Cambodia controlled by communist Khmer Rouge guerrillas and noncommunist forces loyal to Cambodian ex-premier Son Sann. Both groups and a third faction headed by former chief of state Prince Norodom Sihanouk are separately battling the Vietnamese occupation of their country.

The scope of the offensive and its relatively heavy cost in casualties had led some military observers here to believe the Vietnamese might try to hold onto some of the territory they seized. However, in recent weeks Vietnamese forces have withdrawn from forward positions taken from the Khmer Rouge in the hills of Phnom Malai and Khao Din south of the Thai border town of Aranyaprathet and have given up territory around the village of Samrong in northwestern Cambodia, according to Western diplomats who monitor the country.

However, the diplomats said, the Vietnamese still pose a threat to guerrilla settlements north of Aranyaprathet and are capable of making another cross-border incursion there as they did during the rainy season in June 1980.

The Vietnamese also have pulled out of a cluster of four villages collectively known as Sokh Sann inhabited by about 9,000 supporters of the Khmer People's National Liberation Front led by Son Sann, the diplomats said. The front's estimated 2,000 to 3,000 guerrillas in the area have been returning to the villages, which were heavily damaged when the Vietnamese overran them in mid-March, the sources said.

They said the front was trying to make repairs and cultivate ravaged crop fields but had not yet reoccupied the villages located in southwestern Cambodia's Cardamom Mountain range. Most of the civilian inhabitants remain encamped on the Thai side of the border near the Thai town of Trat.

The Vietnamese offensive apparently suffered from logistical failures, widespread malaria and relatively early rains marking the end of the dry season, which usually lasts from November to May.

"It was not a very successful campaign," a U.S. diplomat said. "The Khmer Rouge have apparently withstood what the Vietnamese could throw at them. They can feel fairly satisfied with this dry season; they held."

The Vietnamese offensive was aimed mainly at engaging the enemy, much as U.S. forces tried to do in Vietnam, this diplomat said. "It was essentially a search-and-destroy effort," he said.

The Khmer Rouge sustained heavier casualties than in past Vietnamese offensives, but not enough to damage seriously the estimated 30,000- to 40,000-strong guerrilla force, diplomats said. While there are no reliable casualty estimates for either side during the fighting, diplomats believe they number in the thousands, with the Vietnamese hardest hit. According to one intelligence report, Vietnamese casualties, including malaria victims, increased 30 percent from last year's dry season.

"The Vietnamese losses have been very heavy," said squadron leader Prasong Soonsiri, the head of Thailand's National Security Council. He told reporters recently that many Vietnamese soldiers were wounded by Khmer Rouge land mines and that Hanoi's troops also were suffering acute food shortages.

The results of the Vietnamese offensive underscore that a military stalemate persists in Cambodia, officials said.

"The Vietnamese control most of the population, but they're incapable of rooting out the Khmer Rouge," one diplomat said. "It's an expensive waiting game they're playing."

Yet, according to a Western diplomat who recently visited Hanoi, the Vietnamese leadership appears satisfied with developments in Cambodia.

"There's absolutely no indication the Vietnamese have any reservations about their current course," he said. "They're confident that if they just dig in and outwait everyone else, the world will eventually come around" and accept the Vietnamese-installed government in Phnom Penh.

In this diplomat's view, Hanoi is betting that its estimated 200,000 troops in Cambodia can wear down the Khmer Rouge faster than the group can replace its casualties.