The Soviet Union cautioned China yesterday against "overstepping the forbidden line" in Vietnam, adding another element of strain to an increasingly volatile situation on the Sino-Vietnamese border.

The warning in the journal New Times, while not carrying the full weight of a government pronouncement, appeared to reflect Soviet concern about a substantial Chinese military buildup on the border with Vietnam.

U.S. sources here said the Russians also have increased their naval presence in, or near, Vietnamese waters. Apart from symbolizing support for Hanoi, the deployment of about 15 vessels is believed to be primarily designed to improve Soviet intelligence gathering operations in the area.

The Soviet warning coincided with reports of a continued Chinese military buildup. Japanese military sources were quoted as saying yesterday that "hundreds" of Chinese warplanes have been transferred from central China to the Vietnam border area.

U.S. officials said they are aware of a "significant increase in quality and quantity" of Chinese land forces in the border region. An estimated 100,000 troops are believed to have been stationed there since the buildup began around Jan. 15.

Analysts here said the Chinese may be deploying their troops in "a threatening mode" to get the Hanoi government to withdraw some of the estimated 100,000 Vietnamese troops used to overthrow the Pol Pot government in Cambodia last month.

But these analysts do not rule out the possibility that China may take quick punitive measures against Hanoi.Chinese Vice Premier Teng Hsiao-ping, who returned to Peking yesterday after his American tour, said during a Tokyo stopover Wednesday that "sanctions" against Vietnam were necessary to discourage Soviet expansion in Southeast Asia.

Teng told Japanese legislators that China would act "prudently," but added with emphasis, "Once we say we will do something, we will do it."

The Soviet Union and Vietnam signed a treaty of friendship and cooperation last November that calls for consultations if either side is attacked. The treaty does not provide for "coming to the aid" of the attacked partner, however.

The Soviet magazine article said the collapse of the Cambodian government of Pol Pot was a blow to Peking, Pol Pot's sole ally.

"Now the snubbed dragon wants to show its claws," the article said.

"International public opinion can do much by a timely warning to Peking to prevent it from overstepping the forbidden line," the article said.

Analysts here believe that neither China nor the Soviet Union would like to get involved in a protracted struggle in Vietnam. Teng, who did not elaborate on what "sanctions" against Vietnam he had in mind, told Japanese officials Wednesday that his government "is considering taking an appropriate counteraction even if such an action involves some risk."

In Cambodia, forces loyal to Pol Pot were reported to be fighting Vietnamese-led forces of the new government in Phnom Penh on several fronts. A Khmer Rouge radio, believed to be located in China, said yesterday that Vietnam has sent substantial reinforcements into Cambodia because of continued guerrilla struggle.

Broadcasts emanating from Peking and Hanoi yesterday reported continued skirmishes along the Sino-Vietnamese border. Peking said four Chinese soldiers were killed and three others injured in skirmishes with Vietnamese troops that crossed into China. Hanoi said the Chinese entered Vietnam and killed three Vietnamese border guards.

The transfer of Chinese aircraft to the border area involved relatively old Mig17 and Mig19 jets, the Japanese newspaper Yomiuri said quoting a high-ranking Japanese defense official.

U.S. officials said the deployment of Soviet vessels in Vietnamese waters involved two large warships and a small task force including intelligence gathering and support craft. There was speculation here that the Russians may be assembling a large force needed to prevent China from resupplying the Pol Pot forces in Cambodia.

Such an effect, however, would require a large number of vessels to block off the Gulf of Thailand and access to the Cambodian coast.