The United States sought yesterday to prevent China's incursion into Vietnam from erupting into a wider war by issuing appeals to the two Asian nations to pull their troops back behind their own borders and by cautioning the Soviet Union against direct involvement in the coflict.

Clearly concerned about the impact on Soviet-American relations, which have been through a troubled week, the State Department warned the Russians that "steps which would extend the conflict would have serious consequences" on world peace.

The cautionary statement was delivered by State Department spokesman Hodding Carter, as was the word to reporters that the Carter administration was working to persuade China to halt the long-anticipated invasion and to withdraw its troops from Vietnam.

Implying that the administration believes China has mounted a raid to punish Vietnam for invading and occupying Cambodia in January, Carter added that Vietnam should also withdraw its troops from Cambodia.

Diplomatic efforts to contain the conflict will center on the United Nations, where the United States is prepared to insist that any final resolution link withdrawal by the Chinese from Vietnam with withdrawal by the Vietnamese from Cambodia.

While stressing that their information on the actual fighting is sketchy at this point, military and intelligence officials said the administration views the Chinese action as a rapid thrust into Vietnam's border area rather than as an attempt to march on and occupy Hanoi.

China's objectives may be limited to "straightening out" disputed parts of its Vietnamese border and teaching the Vietnamese a lesson for disregarding its warnings not to go into Cambodia, said a senior official who feels yesterday's attack may turn out to be modeled on the quick Chinese campaign in and out of India in 1962.

Initial Soviet responses have been cautious and suggest that the Kremlin has not yet decided on a course of action, intelligence officials said. Beyond stepping up reconnaissance flights on the Chinese border and keeping electronic intelligence-gathering ships positioned off Vietnam, the Russians have "done nothing pointedly visible," a knowledgeable official said.

Administration spokesmen stressed that President Carter was following the situation closely although he chose to stay at Camp David. An absence of visible high-level policy meetings in the capital was seen by some officials as a tacit admission of Washington's lack of leverage in the conflict between two Asian communist nations over a third.

But administration officials did not mask their concern over the conflict's far-reaching and potentially explosive repercussions on the triangular Soviet-American-Chinese balance, which has been shifting rapidly since the signing in November of a mutual assistance pact between Hanoi and Moscow.

That 25-year treaty obligates the two nations to take "appropriate and effective measures" if either is attacked. U.S. officials said yesterday they were unsure if the Soviet Union would feel obligated to respond forcefully.

In the weeks that followed the pact signing, Vietnam built up its military forces along the Cambodian border and stepped up propaganda attacks on the Khmer Rouge government of Pol Pot. At the same time, China and the United States greatly accelerated negotiations on establishing full diplomatic relations, and announced in mid-December they would do so. Vietnam's invasion of Cambodia came three weeks later.

Hodding Carter laid heavy emphasis yesterday on what he described as the administration's repeated efforts since September to prevent the conflict from erupting into a regional war. His remarks also sought to dispel suggestions that the United States was aligning with China against the Soviet Union.

"We have been in contact with the Chinese, the Vietnamese and the Soviet Union urging and counseling restraint," Carter said, and such contacts are continuing. "We expect the matter to arise at the United Nations."

The Carter administration delivered separate sharply worded protests to the Soviet Union last week over the apparent involvement of Soviet advisers in an incident that led to the killing of U.S. Ambasador Adolph Dubs in Kabul, Afghanistan, and over Soviet media attacks on the U.S. role in Iran.

President Carter "made it clear" to Chinese Vice Premier Teng Hsaioping during their talks here last month that "the United States opposed any further military action in the region," Hodding Carter said, indicating that the Chinese leader had spoken then of "unspecified types of action" against Vietnam.

Asking not to be identified, a State Department official said later that normalization of relations with China is now "an accomplished fact, and will not be reversed."

Declining to say if the administration had any response from the Chinese or Vietnamese to the new calls for restraint and mutual withdrawals, Hodding Carter did not indicate that the administration had any reason to hope that the calls would be successful.

The administration is known to have had previous contacts with Hanoi about withdrawing from Cambodia.

U.S. military officials said they had no indication that the Chinese invasion had been coordinated with Pol Pot's guerrilla forces.

One clue to Chinese intentions is a lack of any large-scale buildup of supplies that would be needed to support a sustained invasion, U.S. officials said.

Responding to questions, Hodding Carter said that the administration was not attempting to equate the invasion of Cambodia by Vietnam to the action taken by the Chinese yesterday. But by linking the two in his statements, he indirectly drew attention to Chinese assertions that they would only be "punishing" the Vietnamese by whatever military action they decided to take.