The Senate Foreign Relations Committee voted unanimously yesterday urge Americans not to participate in the 1980 Summer Olympic Games if they are not moved from Moscow in response to the Soviet invasion of Afghanistan.

In a resolution expected to be approved by the Senate today, the committee also called for permanent sites for the Games to avoid what committee Chairman Frank Church (D-Idaho) called the "movable propaganda feast" staged every four years by host countries. Greece, where the games originated, was proposed as one permanent location.

The resolution, which generally parallels one adopted 386 to 12 last week by the House, calls on the International Olympic Committee (IOC) to relocate, postpone or cancel the Moscow Games -- as requested last weekend by the U.S. Olympic Committee (USOC) under prodding from the Carter administration.

If the IOC refuses, the committee urged that "no American team should participate in those Games and no American should attend them in any capacity." But it stopped short of urging any coercive action to back up the boycott, such as passport restrictions in the event U.S. athletes tried to participate. It also called on the secretary of state to "intensify efforts" to rally other nations behind the American position.

In testimony before the committee, Deputy Secretary of State Warren Christopher said he expected at least 20 nations, in what he called a "conservative" estimate, to support the USOC position.

USOC Executive Director F. Don Miller reiterated that he understands that both Japan and West Germany support removal of the Games from Moscow, although officials of the West German Olympic Committee denied yesterday that they have taken such a position. A Japanese Olympic official earlier denied that his committee had done so.

Christopher said the United States has received encouraging signals from Africa, South America and Asia as well as Europe. However, Miller said no national Olympic committee has yet proposed a boycott and many have publicly opposed such action. Although Carter has proposed that the United States not participate if Soviet troops are not withdrawn from Afghanistan by Feb. 20, the USOC, at its executive board meeting last weekend in Colorado, deferred action on that issue until the IOC acts on the USOC's resolution.

(The two Americans on the 89-member IOC said yesterday that there was virtually no chance that the USOC proposal would be favorably received, and an IOC source at the organization's headquarters in Lauanne, Switzerland, said that relocation of the Games would be "against IOC rules," and that postposing them is "absolutely impossible." (Details on Page D1)

The Senate committee's call for an Olympic pullout from Moscow was not conditioned on Soviet troop withdrawal. Any withdrawal deadline would only encourage the Soviets to "finish [their] dirty work" expeditiously, Sen. David H. Pryor (D-Ark.) said in uging that the committee drop troop withdrawal language from its resolution.

The proposal for a permanent site for the Games came from Sen. Bill Bradley (D-N.J.), a former professional basketball player and member of the U.S. team that won a gold medal in the 1964 Tokyo Olympics.

Bradley, who wrote an article four years ago predicting that the 1980 Olympics would succumb to Soviet American hostilities, supported U.S. withdrawal from the Moscow Games as well as establishment of a permanent Olympics "home" in Greece.

"By withdrawing from the Moscow games ... we will show not only that we condemn [the Afghanistan] invasion but also that we no longer will participate in the political corruption of the Olympics," said Bradley, the only former Olympic athlete currently in Congress.

Of three athletes who testified before the Senate committee, one Anita Defrantz, captain of the 1976 U.S. women's rowing team and a contender for the 1980 Games, complained that athletes now in training for the Olympics are being "portrayed as villains" if they want to compete. Said Defrantz: "Please. if you must use us ... don't destroy us." In response, the committee called in its resolution for "continued support, commendation and contributions for athletes training for the Olympics.

Several of the senators who testified said they had overcome intial misgivings about the effectiveness of a boycott. Only Ted Stevens (R-Alaska) opposed a boycott resolution, saying the decision should be left to the athletes.

Willi Daume, president of the West German Olympic Committee, yesterday denied published reports that he had assured the USOC's Miller that the German committee would join in petitioning the IOC to move, cancel or postpone the Moscow Games.

Miller said he talked with Daume by telephone over the weekend and was told that the West German committee "was considering a resolution consistent with the one we passed, and that he felt it would be adopted."

Miller said that he understood that the German resolution would be "similar to the USOC's, particularly with reference to the part encouraging their athletes to continue to train."

The USOC resolved to "continue to select and prepare the United States Olympic team, whether or not the United States competes in the Summer Games of 1980, in order to recognize the athletes who have been training as Olympians."