The Soviet Union, in an apparent expansion of its military buildup off Japan's northern coast, has established a new facility on one of the islands long claimed by Tokyo, government sources said here today.

They also indicated that the Soviet troop commitment has grown considerable since last winter when Japan protested and demanded that the islands be returned to its control.

The new facility was discovered recently on the island of Shikotan, one of the four islands retained by the Soviet Union since the end of World War II despite Japan's objections.

It was disclosed last year that the Soviets have built airstrips, barracks, missle sites and other facilities on two of those islands -- Etorofu and Kunashiri. The latest report means that there are Soviet installations of some type on three of the four islands, which lie only a few miles off the coast of Japan's northernmost island of Hokkaido.

The nature of the new Soviet facility, which is said to have been discovered by U.S. spy satellites, was not clear. But the Soviet buildup coincides with a period of strain in Soviet Japanese relations that came following Japan's signing of a friendship and cooperation treaty with China, Moscow's arch rival in the Communist world.

Today's disclosure also comes on the eve of high-level Sino-Soviet talks in Moscow, the first attempt in 10 years to seek normalization of relations between the two countries. It was unclear what effect the Soviet move would have on the delicate balance between the three Asian powers.

Japanese assessment of Soviet motives in deploying troops on the disputed islands varied. Some sources said the motive was primarily political and designed to reinforce the Soviet claim on the islands. Others said Soviet installations there are primarily for intelligence-gathering missions directed against the United States.

The Japanese Foreign Ministry confirmed only in general terms that the new facility had been spotted by American intelligence satellites. A spokesman said it appeared to be a group of tents and declined to say whether there was information to prove that the Soviets had moved troops or equipment to Shikotan.

He said there was no indication that an airstrip is being constructed, as on the other islands, and he carefully refrained from referring to the facility as a military base, saying that more investigation is needed to determine just what is there.

The Foreign Ministry was inclined to play down the significance of the latest expansion, and in New York Foreign Minister Sunao Sonoda said it would not be in the interests of Japan to protest the latest development too much.

But is also was pointed out here that extending the military commitment to Shikotan carried with it a new political element in the continuing argument over the islands' ownership.

Japan has repeatedly claimed control of all four islands. The Soviets have rejected that claim but in 1956 promised that eventually the two smaller ones, Shikotan and Habomai, would be returned to Japan.

The condition for that was a Japanese promise to sign a peace and friendship treaty. Japan has refused to sign such a treaty until all four islands are returned.

To some observers, the latest expansion represented a political backtracking by the Soviets. They are fortifying an island they once tacitly agreed should be returned to Japan some day.

Japanese defense officials also indicated that in recent months the Soviets may have increased their forces considerable on the islands.

One source said that Soviet strength on Etorofu, Kunashiri and Shikotan is "nearing the division class." That would normally mean a commitment of 10,000 to 12,000 troops, but the source observed that not all Soviet divisions are maintained at full strength.

In February, defense officials here estimated that the Soviets had stationed between 3,000 and 4,000 troops on Etorofu and Kunashiri, including border guards. If it is true that nearly a division is now stationed there, it would mean that the Soviets have returned to the approximate force strength deployed on those islands in the immediate post-war period. Most were removed in 1960.

The defense source also said that a considerable amount of "very good" military equipment has been placed on the islands.

Prime Minister Massayoshi Ohira told reporters that he was aware of the reports on Shikotan but declined to comment. However, his chief cabinet secretary, Rokusuke Tanaka, said the government would lodge protests with the Soviet Union if the initial reports were confirmed.