While loudly denouncing the U.S. boycott of the Moscow Olympics as "political," the Soviet Union quietly withdrew in January from this year's international Math Olympiad -- and for reasons that some U.S. observers believe may be highly political.

Some believe the Soviets, taken aback by good showings in recent years by the United States and West Germany, want an extra year to train their competitors to make sure thay make good showing in the annual contest.

Others speculate the Soviets are irritated at invitations to the People's Republic of China, which hasn't shown up so far but which may come to the 1981 Olympiad, expected to be held at Georgetown University.

The Soviets themselves never gave a reason for withdrawing.

Whatever it is, the Soviet withdrawal from the 1980 Olympiad has cast a doubt on whether the competition will be held this year. Outer Mongolia, a Soviet satellite which was being considered as host country, has notified Olympiad officials they can't do it. So far, none of the Eastern European countries in the Soviet orbit has sent in word that it is going to compete.

Dr. Samuel Greitzer, Rutgers University professor emeritus and coach-organizer of the U.S. team, said in a phone interview that he is hoping that a host country can be found and that most of the countries invited will participate. He said he doesn't know the reason for Soviet withdrawal and he declined to speculate.

The International Mathematical Olympiad was founded in 1959 by the Romanians. The United States joined in 1974. Each team consists of eight students, usually high-school seniors.

In practice, the contestants are the best young mathematicians in each country. Greitzer picks the eight U.S. contestants from the top finishers in a national competition, called the U.S.A. Mathemetical Olympiad, involving 100 of the best math students in the country.

For years the Soviets have been the powerhouse in the international competition, winning three times in the last six contests.

The United States wasn't expected to do well. It doesn't prime students at length especially for the competition. Moreover, math education here places less emphasis on broad-spectrum training than in Europe.

However, the United States astonished the math world by finished second in 1974, third in 1975 and 1976, first in 1977 and second in 1978, although it dropped to fifth last year.

In 1978, the Russians indicated they would like to hold the competition every two years instead of annually. But Romania refused, and held the competition anyway. It also invited China, though the Chinese didn't participate. Russia withdrew from the 1978 Olympia. East Germany and Hungary followed suit.

In 1979 Russia came back, apparently after it was clear China wouldn't participate that year. But in January 1980, Greitzer received a letter from his Russian counterparts declaring without explanation that they wouldn't come to the 1980 games and that a decision on 1981 would be up to the Ministry of Education.

One mathematician here said. "The Russians are playing the game. They train their people in two-year cycles in order to win. So they just want games every two years." Others said annoyance with the invitation to China might well be a factor.

A government diplomatic source has said, "The Soviets have repeatedly used sports and competition for political purposes -- blackballing a country like Israel, for example, from a competition. This would be entirely consistent with their behavior.