Cambodia's revolutionary government, reviled in the West for alleged mass killings and threatened by a huge Vietnamese army, has begun to emerge from its diplomatic shell in a desparate bid for survival.

The mysterious, xenophobic Communist leaders who took power in Phnom Penh in 1975 have moved to normalize relations with their non-Communist neighbors, sought economic aid from Japan and appealed to the thousands of Cambodians who have fled abroad to return and help rebuild the country.

Cambodia's now well-traveled deputy prime minister, Ieng Sary, has completed a well-publicized appearance at the non-aligned conference in Belgrade and is expected to travel next month to one of Asia's most pro-Western nations, the Philippines, sources in Manila said.

"They are definitely coming out of their shell," said one Indochina expert here. "The Cambodians don't have much pull, they have very limited resources and not a very good reputation, so they seem so be coming to realize they need as many friends as they can get."

Whether any of this activity will help the Phnom Penh leadership stave off severe economic difficulties and a combined invasion and insurgency organized by their former ally, Vietnam, remains to be seen.

The secretive Cambodian Communist Party clique led by Prime Minister Pol Pot is already the target of an insurgency movement of disenchanted Cambodian Communists organized by Hanoi in Cambodia's Krau region in the northeast, where Vietnamese troops also are awaiting the end of the rainy season. Pol Pot may be even more vulnerable to intrigue by officials close to him who see a way to oust him and please Cambodia's number one ally, China.

Much of Cambodia's renewed diplomatic activity seems to be the result of tutoring from Peking, which has won much international praise and support through its own diplomatic revival in recent years.

When the pro-American Cambodian government collapsed in 1975, the Pol Pot group moved Cambodian city dwellers out into the countryside. It imposed a primitive, barter economy and a political purge that, according to refugee accounts, brought hundreds of thousands of deaths by murder, disease and starvation. This left Phnom Penh with few friends in the world besides China, which saw a need to stymie efforts to turn Cambodia into a client state of Vietnam, and China's arch rival, the Soviet Union.

For a long period, little was heard from Cambodia on the international scene. Diplomats from Phnom Penh usually traveled only to Peking. But in the last year, as their border conflict with Vietnam has exploded into fullscale war, the Cambodians have begun to seek help almost frantically from other quarters in apparent hopes of showing the Vietnamese they are not dealing with a weak international outcast.

Ieng Sary, who is responsible for foreign affairs, visited Thailand in July to work out an agreement: that would end violent incidents along the Thai border and allow Phnom Penh to concentrate on the real war with the thousands of Vietnamese troops already reported inside Cambodia. The Cambodian official lamely referred to bloody Cambodian killings of Thai villagers as "misunderstandings" although such incidents seem to have been curtailed recently.

Agreements were reached to normalize relations with Malaysia and Indonesia. An official Singapore delegation visited Phnom Penh and signed an agreement in May restoring economic and trade ties as well as sea transportation and telephone links. Cambodia is seeking a barter exchange of rice for manufactured goods. Ieng Sary also sought aid in Tokyo, although without any immediate success.

China, North Korea, Cuba, Albania, Yugoslavia, Laos, Romania and Egypt have embassies in Phnom Penh, although the ambassadors of Yugoslavia, Romania and Egypt reside in Peking. Japan's ambassador to Peking is also accedited to Phnom Penh, although the Japanese have no diplomatic staff inside Cambodia. Cambodia has embassies in China, North Korea and Laos, but appears to have neither the funds nor the trained diplomats to staff much more than that.

Perhaps the country's most serious international diplomatic problem remains the worldwide outcry generated by reports from Cambodian refugees of mass torture and executions. President Carter has labeled Phnom Penh the world's worst violator of human rights, and Britain took the reports of Cambodian atrocities to the United Nations Human Rights Commission, leading to a Cambodian reply that British citizens only enjoyed the right to be slaves, thieves, prostitutes or unemployed.

Lately, the Phnom Penh government's efforts to improve its international reputation have been more sophisticated. It has used Vietnam's invasion of its territory to good advantage, inviting friendly correspondents, such as two reporters from pro-Peking newspapers in Hong Kong, to see small Cambodian units bravely fending off the Vietnamese onslaughts.

Pol Pot, one of the most publicity shy leaders in the world, has granted lengthy interviews to friendly visitors such as a Yugoslav press delegation and the Belgian-Cambodian friendship association. He has tried to steer the conversation to his plans to rebuild the economy and increase the population, now believed to be about six or seven million. He has shrugged off the reports of atrocities, acknowledging that a few landlords and rebels working for the Vietnamese or the Americans had to be killed.

According to Phnom Penh radio, the premier has invited all Cambodians living abroad, including those who fled, to return home "without fear." He said Cambodian exiles remain abroad because "they have been fooled by our enemies."

"If they want to return, no matter who they are, we will give them a cordial reception." he said. More than 16,000 Cambodian refugees are living in Thailand and tens of thousands have been resettled in France, the United States and Australia.

Many diplomatic observes here are betting, however, that the present Phnom Penh government, or at least the small group around Pol Pot, will not be able to withstand general dissatisfaction within the country and the expected new attack by Vietnam.

Hanoi launched invasions in December and June that the Cambodians, well armed and trained by the Chinese, managed to blunt.But the Vietnamese have reportedly reinforced their border troops to a total of about 120,000 men, against a total estimated Cambodian strength of 90,000 and are just waiting for the rainy season to end. Heavy Vietnamese air strikes have been reported, although not confirmed, even during the rainy season.

At least as serious is the insurgency led by Cambodians, many of them part of a purged pro-Hanoi faction in the Cambodian party, in the northeastern sections of the country now occupied by the Vietnamese. Some observers argue that Pol Pot could remain bidden in his own sanctuaries and carry on an endless civil war with the pro-Vietnamese elements, but he might still fall at the hands of aides who think his severe methods have gone too far.

Although there is absolutely no concrete evidence of this, some of those close to Pol Pot may be encouraged by the Chinese to overthrown him. Peking has made a point of trying to avoid mass executions in winning popular support of its own administration of China.

"I'm not at all sure the Chinese are 100 percent taken with Pol Pot," said one diplomat.