An unexpected about-face by Prince Norodom Sihanouk, the exiled and unpredictable former Cambodian chief of state, has thrown new confusion into diplomatic efforts by Southeast Asia's noncommunist nations to end Vietnamese occupation of his country.

The question of how to deal with Vietnam, particularly its two-year-old role in Kampuchea, as Cambodia is now known internationally, is considered important to the future of U.S.-Chinese relations as they develop under the new administration in Washington. Against that background, specialists had regarded as something of a breakthrough a pledge from Sihanouk last month to participate in a coalition of anti-Vietnamese Cambodian groups including the Khmer Rouge.

But now, in more recent statements, the mercurial Sihanouk has backed away from these earlier promises, complicating efforts by the Association of Southeast Asia Nations (ASEAN) to convene an international conference on Cambodia under U.N. auspices.

Sihanouk first reemerged as a key figure in the Cambodian issue in February when he came out of a self-described "political retirement" and issued his surprise statement expressing willingness to join the Khmer Rouge in a board front against the Vietnamese. Communist Khmer Rouge forces have been fighting Vietnamese troops in Cambodia since the Vietnamses invaded the country in December 1978 and installed a government under Heng Samrin.

The prince, who lent his prestige to a Khmer Rouge government in 1975 only to be placed under house arrest, indicated earlier this month that he really was not serious about joining a coalition with the Khmer Rouge and preferred a political compromise with Vietnam to continued warfare.

That position has come as somewhat of a setback to ASEAN, the United States and China, who have been hoping that the guerrilla war will eventually push the Vietnamese to the conference table and that Cambodian "nationalist" figures will rally to a united front against the Vietnamese occupation.

Implicit in the common desire of Washington, Peking and the ASEAN countries for a Cambodian coalition is a need to spruce up the image of the Khmer Rouge, who have been blamed for the deaths of 1 million to 3 million Cambodians under the regime of ousted leader Pol Pot.

With the prospects for coalition now in confusion, hopes for a negotiated solution of the Cambodian problem are being pinned on a little-reported visit to Southeast Asia by a special representative of U.N. Secretary General Kurt Waldheim. However, so complex and delicate are the problems involved that even defining the purpose of the purpose of the visit caused a minor diplomatic row at the United Nations last week.

One problem is that although the five ASEAN members -- Thailand, Indonesia, Malaysia, Singapore and the philippines -- strongly advocate an international conference on Cambodia, Waldheim has been relucant to organize one as called for in a General Assembly resolution.

That reluctance led to some wrangling at U.N. headquarters when ASEAN members threatened not to receive Waldheim's special envoy unless his mission were specifically linked to the U.N. conference. asian diplomatic sources said Waldheim's office originally described the envoy's mission as one to work out bilateral consultions between countries of the region on the Cambodian issue. The sources said this resembled the position of Vietnam and its Soviet-Bloc allies, who oppose the U.N. conference.

ASEAN insisted that the envoy discuss the implementation of the October 1980 U.N. resolution, which calls for the withdrawal of foreign troops from Cambodia and U.N.-supervised free elections.

Diplomatic sources said Waldheim privately agreed to such a mandate for his envoy, but resisted describing the mission in detail publicly and wanted to keep the tour low key. The sources said Waldheim apparently is trying not to antagonize either side on this issue because he is coming up for reelection this year.

The envoy, Mohammed Essaafi of Tunisia, left New York Friday and is currently in Thailand. Besides the other ASEAN countries, he is scheduled to visit Vietnam, Laos and Japan.

Some ASEAN members have expressed determination to hold a conference, even if the Vietnamese refuse to attend, as a way to focus attention on the continuing Vietnamese occupation of Cambodia.

So far, U.S. policy has been to follow ASEAN's lead and endorse -- albeit uncomfortably -- the association's support of the Khmer Rouge government (officially known as Democratic Kampuchea) as the legitimate representative of Cambodia at the United Nations. This policy has not changed under the Reagan administration, although some ASEAN members hope Washington's stance on El Salvador will translate into a tougher line against what they see as Vietnamese expansionism.

Some diplomats feel, however, that support for the Khmer Rouge in order to "bleed" the Vietnamese actually plays into Hanoi's hands. According to this argument, the approximately 200,000 Vietnamese troops in Cambodia have not gone all out to mop up Pol Pot's estimated 35,000-strong army because they need its presence in western Cambodia to justify their own occupation.

Although many Cambodians resent the largely Vietnamese-administered government in Phnom Penh and presence of Vietnamese troops in the country, they are said to fear the return of Pol Pot to power even more.

Sihanouk also clearly distrusts the Khmer Rouge. In a recent interview with the Far Eastern Economic Review at his exile palace in Pyongyang, North Korea, he said bluntly: "I do not plan to unite with the Khmer Rouge. I want to stay in the middle."

Explaining his apparent turnabout on that score, Sihanouk said, "I change tactics, but not my mind, not my convictions. And my conviction is that we need a good reconciliation with Vietnam." He said that a united front with the Khmer Rouge was "unrealistic" and that waging war against the Vietnamese was "madness."

Sihanouk also criticized the noncommunist Cambodian resistance, of which a main group is the Khmer People's National Liberation Front led by Son Sann, a former prime minister under the prince.