Cambodia's ousted communist Khmer Rouge rulers have set back efforts to coordinate opposition to the Vietnamese occupation of Cambodia by their rejection of proposals for a loose coalition with two noncommunist resistance groups, according to diplomats here.

The setback came late last week in a letter from Ieng Sary, the Khmer Rouge deputy premier in charge of foreign affairs, to Sinnathamby Rajaratnam, the Singaporean deputy premier who put forward the proposals last November. The proposals later were endorsed by the Association of Southeast Asian Nations (ASEAN), made up of Thailand, Indonesia, Malaysia, Singapore and the Philippines.

The rejection appeared to upset the attempts of Southeast Asia's noncommunist nations to promote a union of anti-Vietnamese Cambodian groups that would allow the noncommunist elements to receive foreign arms aid while glossing over the tarnished image of the Khmer Rouge government. Although internationally vilified for its brutal rule between 1975 and 1979, the Khmer Rouge regime retains U.N. recognition as Cambodia's legitimate government.

According to knowledgeable diplomats, Ieng Sary's letter essentially sets aside the idea for a loose coalition while effusively praising ASEAN's efforts and calling for a "genuine union" under the Khmer Rouge government, which is officially called Democratic Kampuchea.

The letter said any coalition of Cambodian resistance groups "should have well-defined political principles," and it called for a "summit" of resistance leaders at an early date.

The leaders of the three main groups--former Cambodian head of state Prince Norodom Sihanouk; his former premier, Son Sann, and the president of the Khmer Rouge, Khieu Samphan--met for the first time in Singapore last September and agreed to work toward a coalition in the common interest of ending Vietnam's occupation of their country.

An ad hoc committee of the three groups then worked out a governmental structure for a coalition, but Son Sann, 72, withheld his personal participation, fearing that his Khmer People's National Liberation Front would be swallowed up by the more powerful Khmer Rouge.

With the coalition talks deadlocked in November, Singapore proposed a loose coalition that would allow each group to keep its separate identity. Sihanouk and Son Sann accepted the idea, but the Khmer Rouge said it would reply in two months after consulting its followers.

In the interval, frequent Khmer Rouge statements purportedly reflecting the views of the faction's troops and supporters made it clear that Singapore's proposals would be rejected.

Echoing those statements, Ieng Sary's letter dated Jan. 23 said the Khmer Rouge feared that such a loose coalition would not contribute to a strengthened resistance but would "bring about the dislocation of the forces of Democratic Kampuchea both as fighting forces and the legitimate state of Kampuchea."

The letter said this would only benefit Hanoi, which sought to "liquidate the forces of Democratic Kampuchea." The letter further called for a return to the spirit of the Singapore summit in September.

"In a sense we are back where we were then," said a diplomat familiar with Ieng Sary's letter, which has not been made public. He and Thai officials characterized it as a rejection of the ASEAN proposals.

However, Rajaratnam denied yesterday that the letter amounted to a rejection but said he expected Khmer Rouge counterproposals.

China, the main backer of the Khmer Rouge, has not yet reacted officially to the rejection of a loose coalition. But authorities in Peking had indicated earlier that they shared the Khmer Rouge objections to the Singapore proposals.

Chinese Foreign Minister Huang Hua was noncommittal when asked last month by a Thai Foreign Ministry official to persuade the Khmer Rouge to join a coalition, according to Thai officials.

The main objection of the Khmer Rouge, diplomats believe, is that the loose coalition would lower them from the status of a legitimate government to that of just one political party with a share of power in a government. The diplomats said the only inducement for the Khmer Rouge to join would have been Chinese pressure, which evidently was not forthcoming.

The next step for ASEAN, diplomats here said, will be to try to thrash out a common approach to the Khmer Rouge response at a senior officials' meeting early next month. A major question for ASEAN members and their Western backers is whether to funnel military aid to the groups of Sihanouk and Son Sann in the absence of a coalition, the diplomats said.