Vietnamese troops, waging their broadest offensive against resistance groups since invading Cambodia four years ago, today battled to overcome the last defenders at a refugee camp controlled by exiled leader Prince Norodom Sihanouk, and a Thai warplane reportedly bombed Vietnamese positions along the Thai-Cambodian border.

The warplane, identified by Thai reporters at the border as a U.S.-supplied F5E, reportedly made two strikes against Vietnamese troops who had crossed into Thailand after attacking communist Khmer Rouge guerrillas in the Phnom Chat area of western Cambodia. The strikes marked the first time that Thailand has used its Air Force against Vietnamese troops at the border, although spotter planes regularly patrol the embattled frontier.

The Thai Supreme Command in Bangkok refused to confirm or deny the air strikes, but reports from the border said the plane hit small units of Vietnamese troops on the Phnom Pra hill and opposite the former Khmer Rouge camp at Phnom Chat. The reports said the plane drew ineffectual Vietnamese antiaircraft fire.

In Washington, the State Department voiced concern about "the escalating level of violence" and said it "is consulting with the Thai government" on possible U.S. military assistance, United Press International reported.

According to Thai military authorities and western relief officials, the Vietnamese assault on Sihanouk's O Smach settlement on Cambodia's northern border with Thailand forced an additional 20,000 to 25,000 refugees to flee into Thai territory. Relief officials said that since the latest Vietnamese offensive began Thursday, as many as 50,000 Cambodian civilians have crossed the border to escape heavy shelling and tank-led ground assaults.

At least 230 Cambodians, including many women and children, were being treated for war wounds at hospitals on the Thai side of the border, the officials said. They said it was not possible to estimate the number of other casualties.

The attack on O Smach--also known as Sihanouk Borei or Sihanoukville--came as Thai and Vietnamese gunners continued to trade artillery fire across the border at several points.

According to Thai military officials, the heaviest fighting between the two sides took place when a Vietnamese company crossed the border after an assault backed by armor and artillery on the Phnom Chat base Thursday.

The officials said five Thai soldiers were killed and 12 wounded in artillery duels and hand-to-hand fighting that drove the Vietnamese back during the weekend. But other reports from the border said some Vietnamese were still dug in on the slopes of the Phnom Pra hill straddling the frontier.

According to Thailand's Supreme Command, Khmer Rouge guerrillas were continuing to harass the Vietnamese in the Phnom Chat area, whose civilian population has fled into Thailand.

At the same time, a Supreme Command spokesman said, guerrillas belonging to Sihanouk's faction were still fighting Vietnamese attackers at the O Smach camp about 120 miles northeast of Phnom Chat. A U.N. official said clashes were continuing tonight.

The Supreme Command spokesman said the Vietnamese had captured the Sihanouk group's nearby camp of Phnom Saleuth, forcing its 5,000 inhabitants to flee. He denied Thai newspaper reports that Hanoi's forces had already overrun O Smach, although the civilian population had fled and reports from the area indicated that the camp's capture was only a matter of time.

The attack began at dawn yesterday with a fierce artillery barrage, putting thousands of Cambodian civilians to flight. Western relief officials said several thousand of the camp's 30,000 inhabitants remained unaccounted for.

For the Vietnamese, the camp has mainly psychological rather than military value, western diplomats said. It is the headquarters of the Moulinaka guerrilla faction loyal to Sihanouk, who heads a coalition government of Cambodian resistance groups. The other two partners in the uneasy coalition are the anti- communist Khmer People's National Liberation Front led by Son Sann and the communist Khmer Rouge, who were driven from power by the invading Vietnamese in 1979.

While Sihanouk holds the greatest international prestige among the coalition leaders, his group is the weakest militarily. Its estimated 4,000 fighters pose little threat to the 160,000 to 180,000 Vietnamese troops occupying Cambodia.

Far more dangerous foes are the estimated 30,000 hardened guerrillas of the Khmer Rouge, who have borne the brunt of most past Vietnamese offensives.

During the current dry-season fighting, however, the Vietnamese have attacked all three groups in the coalition. The offensive began in January with assaults on positions held by Son Sann's group, which counts 9,000 to 12,000 fighters.

About 30,000 refugees in Son Sann's Nong Chan camp were forced to flee into Thailand in early February, and the group has expressed fears that its Nong Samet and Ban Sangae settlements inhabited by 90,000 Cambodians may come under attack soon.

Vietnam, meanwhile, has blamed the border fighting on the United States, China and Thailand. The official Radio Hanoi said the Heng Samrin government, which Vietnamese troops installed in Phnom Penh in 1979, had the right to attack rebels who it said were seeking the return of the "genocidal" Khmer Rouge regime led by ousted dictator Pol Pot.

"If Thailand had not opened its borders to the Pol Pot remnants and other reactionaries, this situation would never have happened," the radio said.