The first diplomatic talks between warring parties in the nine-year Cambodian conflict have a good chance of beginning before the end of the year, U.S. and Southeast Asian diplomats said yesterday.

The prospects of a parley were outlined to Secretary of State George P. Shultz at the United Nations yesterday morning by Prince Norodom Sihanouk, Cambodia's former ruler, who has initiated the effort, the diplomats said. They said Sihanouk might meet with Hun Sen, the prime minister of Cambodia's Vietnamese-backed government, late next month in France.

An alternative to that meeting would be an informal "cocktail party" session of rival Cambodian factions, also involving representatives of the anti-Vietnamese resistance as well as representatives of the Vietnamese-backed Phnom Penh regime, to be held in November or December in Indonesia.

"This doesn't mean the Kampuchea {Cambodia} problem will be solved," cautioned Thai Foreign Minister Siddhi Savetsila, a leader of the noncommunist Association of Southeast Asian Nations (ASEAN), which has strongly opposed the December 1978 Vietnamese invasion and subsequent occupation. Nonetheless, Siddhi said "everyone feels there is a consensus on the Vietnamese side to have a serious talk about serious solutions" and that ASEAN is backing Sihanouk in his efforts to start the discussions.

The potential break in the Cambodian diplomatic logjam was described by Vietnam's Deputy Foreign Minister Nguyen Dy Nien in a U.N. speech Tuesday as "an opportunity not to be missed" in pursuit of a settlement. The speech made an unprecedented approving mention of Sihanouk, who has headed the anti-Vietnamese coalition, a recognition that diplomats considered significant.

U.S. and ASEAN diplomats said Vietnam's willingness to encourage talks is partly an effort to reduce last year's lopsided 115-vote majority in the General Assembly against the Vietnamese occupation of Cambodia. This year's Cambodia debate and vote at the United Nations is scheduled for Oct. 13-14.

Several diplomats said there also are clear signs that Vietnam is weary of the war in Cambodia, and that Sihanouk's increasingly active role provides the best chance in years to get negotiations started.

State Department spokesman Charles E. Redman said Shultz, in the meeting with Sihanouk, "stressed our very strong support for the freedom and independence of Cambodia, that we fully support Prince Sihanouk in his role in this process of trying to restore freedom and independence."

Sihanouk, 64, was the hereditary and then elected leader of Cambodia from 1941 until he was deposed in 1970. In recent years he has been president of the anti-Vietnamese coalition of resistance groups, including noncommunist forces based along the Thai-Cambodian border and the larger Khmer Rouge resistance army associated with Cambodian revolutionary Pol Pot.

In May, Sihanouk made the surprise announcement that he was taking a year's leave of absence from the leadership of the anti-Vietnamese forces. U.S. officials said he later explained he did this to obtain more freedom of action to bring about negotiations and to distance himself from the Khmer Rouge.

The Khmer Rouge, in a broadcast over their clandestine radio yesterday, distanced themselves from Sihanouk, saying, "The proposed talks among Cambodians, which Vietnam is using to deceive people in its current propaganda, is just a ploy attempting to cover up Vietnam's aggression in Cambodia."

The "cocktail party" formulation of no-conditions talks among the warring Cambodian parties was suggested last July by Indonesian Foreign Minister Mochtar Kusumaatmadja. Mochtar said Vietnam should join these talks "at a later stage," a condition that ASEAN has sought to tighten up.

The plan for Sihanouk to meet Hun Sen, a key leader of the Vietnamese-backed government, was attributed to Cambodian elders living abroad, but many think it was initiated by Sihanouk. He has insisted that their talks be made public.

Hun Sen has not accepted Sihanouk's conditions but said last month for the first time that his regime is willing to have talks with the anti-Vietnamese resistance.