The United States charged yesterday that the military action in Cambodia is "a clear violation" by Vietnam of its neighbor's territorial integrity, and expressecd concern that the confliet in Indochina might escalate into a Sino-Soviet congrontation.

A State Department declaration, personally authorized by Secretary of State Cyrus R. Vance, also called on the Vietnamese to "remove their forces from Cambodia."

U.S. concern that the conflict might spread to the major communist powers was heightened by intelligence reports from Asia. Officials said China appears to be continuing its movement of troops and heavy weapons, including tanks, artillery and combat aircraft, to positions less than 30 miles from the Sino-Vietnamese border.

At the same time,there were reports of Soviet military movements near the Sino-Soviet border, according to official sources. It was not clear whether these movements were connected with the Indochina strife, although there was speculation that the Soviets are warning Peking against any attack on Vietnam, which two months ago became a treaty ally of Moscow.

The fast-paced developments in Indochina and especially the potential involvement of the two communist giants generated anxiety in Washington about the ffect on international stability and power relationships, but there seemed to be little that the United States could do to affect the situation.

"It is not our fight," said a State Department official who explained that the United States seeks to bring the gravity of the problem bnefore the community of nations.

Beyone the public statements, there were indications that the United States was coulseling restraint in diplomatic exchanges with Peking and Moscow.

President Carter, who received the news from Indochina while on a deepsea fishing trip near Guadeloupe in the French West Indies, had no comment except to tell reporters mistakenly that "the Vietcong have taken Phnom Penh." The Vietcong, as they were known in the west, were the Vietnamese communist "liberation" forces in the previous Indochina war.

Despite Vietnam's claim that Phnom Penh has fallen to a force of Cambodian rebels, the State Department statement, conveyed by spokesman John F. Cannon, said "the Vietnamese incursion into Cambodia is a clear violation of the territorial integrity of a sovereign state. We believe the Vietnamese should remove their forces from Cambodia."

Cannot said "all countries both in the region and elsewhere should urge restraint on both parties to the conflict [Vietnam and Cambodia] to work toward a withdrawal of the foreign forces and to avoid any expansion of the geographical area of the conflict."

He added that "one of our chief concerns is that this not escalate into a third-party involvement and confrontation."

The official said the United States supports a United Nations Security Council debate on the conflict, as called for by the Cambodian regime of Pol Pot last week. Vietnam has opposed such a debate, calling it interference in Cambodia's intrnal affairs.

U.N. Ambassador Andrew Young, commenting on the Cambodian situation during a New York press conference, said. "I almost think it's always wrong for a country to transgress the borders of another country, but in the case of Cambodia I'm not terribly upset... It is a country [Cambodia] that has killed so many of its own people I don't know if any American can have a clear opinion on it... It's such a terribly ambiguous moral situation."

Young stressed that he did not intend for his remarks to imply approval of invasion by a foreign country. Speaking of U.S. government policy, he said, "We do not approve of this kind of intervention."

But he also said he did not want to be put in the position or to place himself in the position of doing anything that would seem to condone the alleged human rights abuses of the Pol Pot regime.

State Department officials said there was no indication so far of troop movements from neighboring Thailand, which has expressed apprehension about a Vietnamese-dominated regime in Cambodia. Thai forces, however, have been put on military alert.

There has been speculation that Thai-backed Cambodians might try to occupy a buffer zone near the Thai border in the Western part of Cambodia if the Pol Pot regime fell.

U.S. analysts continue to be uncertain about the purpose of China's buildup near its border with Vietnam, and to express skepticism that a Chinese military strike at Hanoi would affect the situation in Cambodia or otherwise benefit the Chinese.

Nonetheless, it is clear that Chinese prestige is at stake in the fate of their Cambodian allies. The Chinese belief that Vietnam is acting in alliance with the Soviet Union makes the situation an emotional one for Peking and thus to some extent unpredictable, officials said.

Chinese military action against Vietnam would add a new factor to the Washington-Moscow-Peking triangular relations that have come to prominence dramatically since the Carter administration announced on Dec. 15 that it would normalize its diplomatic ties to China.

Such a military move might be particularly difficult for Peking at a time when Vice Premier Teng Hsiao-ping is preparing to visit Washington to advance the improved relationship with the United States.

The Vietnamese Foreign Ministry, in a statement read on Hanoi Radio, charged yesterday that China "again feverishly moved large military forces and large amounts of war material close to border areas and intensified provocations to threaten the Socialist Republic of Vietnam." The statement denounced Chinese "provocations and war preparations" against Vietnam and said Chinese authorities will "bear full responsibility" for the consequences.

Moscow has not yet made a major official pronouncement on the new Cambodian events or the Chinese reaction to them. The Soviet-Vietnamese "friendship and cooperation" treaty signed last Nov. 3 stopped short of being a pact of military alliance, but calls for consultations in case of attack or the threat of attack "for the purpose of eliminating the threat and taking appropriate and effective measures to safeguard peace and security."