An Australian effort to promote talks on a settlement of the conflict in Cambodia has ended in failure amid the latest Vietnamese onslaught against Cambodian resistance guerrillas.

The failure also apparently has dashed Australian hopes of forging a role for itself in settling the six-year-old conflict as part of a more activist, Asia-oriented foreign policy under Foreign Minister William Hayden.

Hayden, who ended an 11-day tour of Malaysia, Laos, Vietnam, Thailand and Singapore yesterday, met the premier of the Hanoi-installed Cambodian government in Ho Chi Minh City last week and brought out what he called fresh "proposals" for ending the war.

Thailand immediately rejected the proposals on grounds that Vietnamese forces had crossed the northern Cambodian border in an attack on a Cambodian resistance base and were fighting Thai troops.

In any case, senior Thai officials said, the proposals were "nothing new" and provided no basis for negotiation since they amounted to demands for the surrender of Cambodian resistance forces and recognition of the Heng Samrin government installed in Phnom Penh by invading Vietnamese troops in January 1979.

The president of a U.N.-recognized three-party resistance coalition, Prince Norodom Sihanouk, also rejected the proposals today as a Vietnamese "trick" to impose a communist regime on Cambodia.

A day after meeting Hun Sen, the Phnom Penh government's premier and foreign minister, at the behest of Vietnamese authorities in Ho Chi Minh City, Hayden said at a press conference in Bangkok Sunday that he had clarified "a number of formulations" important to the search for a solution in Cambodia.

The Phnom Penh government is not recognized by the United Nations or any western country, and Hayden's meeting with Hun Sen angered Thai leaders who saw it as boosting Vietnam's efforts to give its Cambodian clients legitimacy.

Hayden said Hun Sen told him that the demands of the Vietnamese and Heng Samrin governments for the "elimination of Pol Pot and his clique" as a condition for a political solution in Cambodia meant that the former Cambodian dictator and one of his chief aides, Ieng Sary, must stay out of Cambodia. But other leaders and members of the Khmer Rouge guerrilla group were welcome to return "provided they lay down their arms, cease being a military force and are prepared to participate in the internal reconstruction and development" of Cambodia, Hayden quoted Hun Sen as saying.

Hayden said a Vietnamese proposal for free elections in Cambodia meant that "anyone who had joined the mainstream of Cambodian community life could participate, but there would only be room for one party, the Communist Party" of the Heng Samrin government.

On the question of international supervision of elections, Hayden said that the Phnom Penh government "would as a matter of sovereign right and self-respect conduct the elections under its authority, but that outside observers would be welcome" under conditions to be worked out.

The immediate Thai response was that it would not consider any proposals while Vietnamese forces were engaged in combat on Thai soil. Hayden said he came away from a meeting Sunday with Thai Foreign Minister Siddhi Savetsila convinced that Vietnamese troops had crossed into Thailand, despite denials made to him by Vietnamese Premier Pham Van Dong and Foreign Minister Nguyen Co Thach.

In any case, Thai Foreign Ministry officials said, the "so-called proposals" from Vietnam were a far cry from U.N. resolutions calling for a solution based on complete withdrawal of foreign troops, free U.N.-supervised elections and Cambodia's adoption of a neutral and nonaligned status.

In a news conference today in Surin in northeastern Thailand, Sihanouk called the Vietnamese "dishonest and arrogant." He insisted that resistance forces would continue fighting despite the loss Monday of the last of about 20 bases on the Cambodian side of the 450-mile border with Thailand during Vietnam's current dry-season offensive.

"What they want is for us to join them in the Heng Samrin regime," Sihanouk said. "Vietnam wants an election in Cambodia to be controlled by Heng Samrin. We cannot accept such a condition to impose a communist regime on our people."

Sihanouk said that although his resistance faction had lost its main stronghold, it continues to operate inside Cambodia. "We still have some remote villages under our control," he said.