Secretary of State Cyrus R. Vance's stern political speech at the opening of the International Olympic Committee (IOC) session here Saturday night may have hardened some IOC members' resolve that the Olympic Games should go on in Moscow as scheduled this summer, but other members apparently are wavering.

IOC Director Monique Berlioux said that members were shocked by Vance's tough remarks on what is usually a strictly ceremonial occasion, and some were clearly indignant.

But other members evidently think the prospect of a U.S. withdrawal if the Games are not moved from Moscow, postponed or canceled in response to the Soviet invasion of Afghanistan is serious enough that the matter should be studied further and a decision deferred, pending international developments.

The Associated Press quoted Juan Antonio Samaranch, the Spanish ambassador to the Soviet Union who is a member of the IOC Executive Board and the organization's chief of protocol, as saying, "Some members are considering putting the matter off for a month or two, if that is the best way to save the Games."

Meanwhile, the U.S. government position that countries should decline invitations to participate in the Games unless the IOC moves them to another site received support from a leading member of the West German Olympic Committee and from German Foreign Minister Hans Dietrich Genscher, who said in Bonn: "We expect solidarity from the United States in Berlin, and we will not deny it in the question of the Olympics."

Genscher's comment amplified a statement by German Defense Minister Hans Apel, who said in Munich on Saturday that it would be unthinkable for West German athletes to march into Moscow's Lenin Central Stadium this summer if Americans were not participating.

United Press International, quoting the mass circulation tabloid Bild am Sonntag, reported that Willi Weyer, and influential member of West Germany's Olympic committee, is urging postponement of the Games.

West Germany is considered a nation of pivotal importance in the State Department's efforts to line up Allied and Third World support for the U.S. government's position on the Olympics.

The West German government appears to be moving closer to a full endorsement of the U.S. position. But IOC member Willi Daume, president of the West German Olympic Committee, has said that his group will decide independently whether or not to send athletes to Moscow and will not bow to government pressure.

"This speech by an American politician will only serve to unite the IOC," Daume said after listening to Vance's speech, which was received in frosty silence by most IOC members, "We do not like politics being brought into the Olympic Games in this way."

IOC President Lord Killanin of Ireland made much the same point in his welcoming speech, saying, "I have never denied or ignored the intrusion of politics into the Olympic movement, and I believe it to be in all our interests that these intrusions be resisted."

Killanin recalled that when the 1980 Summer Games were awarded to Moscow in 1974, and an agreement signed, the decision was "welcomed as a symbol of mutual understanding.

"Sadly, the current political situation is different today, but the IOC entered into agreements in 1974 which must, by my mind, be honored by us all."

Vance, without ever mentioning the Soviet Union by name, made clear the U.S. government position that it would be a violation of Olympic principles "to conduct or atend Olympic Games in a nation which is currently engaging in an aggressive war, and has refused to comply with the world community's demand to halt its aggression and withdraw its forces."

The Soviet news agency Tass, in a dispatch from Lake Placid, sharply criticized Vance's "Cold War phraseology" and "provocative demands to move the Olympics from Moscow or cancel them altogether" and called the speech "another example of crude political interference in the affairs of the IOC."

Berlioux, the salaried executive director of the Ioc, said today that "this is the first time in the history of the IOC that a political speech has been made at the opening of a session," and "it was clear that the IOC did not appreciate this departure from tradition because "the Secretary of State did not get the applause he might have gotten with another type of speech."

She also pointed out that Vance neglected to declare the session open -- ostensibly the sole reason for his visit -- and that Lord Killanin performed this formality today.

Killanin, having seen an advance text of Vance's speech, reportedly tried to dissuade him from delivering it, but was told this was impossible because the State Department already had released the text in Washington.

Killanin then reportedly called IOC member Vitaly Smirnov to a meeting and pleaded with him not to walk out during Vance's speech. The two Soviet IOC members attended the opening, but the delegation from the Moscow Olympic Organizing Committee refused to attend after seeing the speech Vance was to give.

Some IOC members reportedly considered walking out in protest during Vance's remarks, but were persuaded not to do so by Samaranch.

Robert J. Kane, president of the U.S. Olympic Committee (USOC), will appear before the full session of the IOC on Monday morning and will make the same strongly-worded case for stripping Moscow of the Games that he presented to the Executive Board on Friday.

The Moscow Olympic Organizing Committee also will make its detailed report on preparations for this summer's Games on Monday, so it is unlikely that any action will be taken on the USOC proposal until Tuesday or announced before Killanin's scheduled press conference Tuesday evening.

"Certainly Lord Killanin will make an announcement. Whether it will be definite or not definite, I don't know, he does not know either. When we have listened to the USOC tomorrow, then the members will debate that problem."

Douglas F. Roby, one of the two American members of the IOC, said he still thinks the IOC will "flatly reject" the U.S. proposal at this time, but will leave open the option of reconsidering it at a later date, either in emergency session of the full IOC or by authorizing the Executive Board act on the matter at its regular meeting in Lausanne, Switzerland, on April 23.

The Associated Press reported that the South American IOC members were in the the forefront of the movement to delay action until the progress of political events can be reviewed.

The Carter administration has set Feb. 20 as the deadline for Soviet troops to be "fully withdrawn" from Afghanistan.After that, the administration has said, it will ask the USOC for a prompt decision -- probably by the first week in March -- on declining the invitation to participate in Moscow.

The administration considers an early and decisive U.S. commitment to withdrawing from the Games important to lining up international support Most national Olympic committees, including the USOC, have expressed the desire to keep their options open until May 24, the deadline for accepting invitations.

Asked if the IOC would consider postponing or canceling the Moscow Games if a substantial number of Western and Third World nations indicated in upcoming months that they would not participate in Moscow, Berlioux said there was no provision in the IOC rules or the letter agreement with Moscow for action on these grounds.

She also reiterated IOC fears that a U.S.-led walkout from the Moscow games could lead to retaliation by the Soviet bloc or other countries against the 1984 Summer Games scheduled in Los Angeles. "If you start this thing you never know," she said. "It should not be envisaged, because it is the best way to kill the Olympic movement and the games. We should try to unite, and not disunite."

Berlioux also took a dim view, as IOC members have, of U.S. government suggestions that "alternative games" be organized for athletes who do not participate at Moscow."Any other games would not be sanctioned by the International Federations (of each sport), so I don't know how they could be organized," she said.

In other business, the IOC terminated the membership of Prince Gholam Reza Pahlavi of Iran, brother of the deposed Shah, who now lives in England, confirmed recognition of new national Olympic committees in Angola, Bangladesh, Laos, Mauritania and Seychelles; gave provisional recognition to committees in Mozambique and Vietnam, and took under study a Greek proposal that Greece become the Permanent site of the Summer Games.

In a related development, Kuwait announced today that it will send athletes to the Moscow Games this summer. Malaysia, however, has decided against participating.