Carter administration and U.S. Olympic Committee officials predicted today that many major free world countries will soon follow the USOC's lead and decide not to participate in this summer's Olympic Games in Moscow.

The momentum for a widespread boycott to punish the Soviet Union for its invasion of Afghanistan appears to have been revived by the 1,604-to-797 vote Saturday by the USOC's House of Delegates here in support of President Carter's directive that no U.S. team compete in Moscow.

The USOC decision -- which, in an ironic twist, came after an impassioned speech in support of Carter and the boycott by William E. Simon, the former U.S. treasurey secretary who is a leading Ronald Reagan campaigner and Carter administration critic -- is expected to become a rallying point for expanding the boycott internationally.

"I think the USOC's decision is a critically important element, a major step in our efforts to have other nations go along with the boycott," said White House counsel Lloyd Cutler.

"The United States is the other greater superpower and the leader of the free world alliance," Cutler said. "We have every reason to hope that, in light of the USOC's decision, other governments and other Olympic committees -- including most of the major political or economic or sports powers outside the Soviet bloc -- will in good time decide to join this movement and will not send teams to Moscow."

Australian Prime Minister Malcolm Fraser said in Canberra that the U.S. decision strengthened the boycott effort and he hoped the Australian Olympic Federation would vote to boycott at a meeting later this week.

In Hague, Dutch Olympic Chairman Jacobus Idenburg said his ruling body might change its position as a result of the U.S. move and decide not to send a team to Moscow.

Informed sources in Moscow suggested the Japanese may send only a partial team as a result of the U.S. decision.

The Olympic movement's executives are to hold a special meeting in Lausanne, Switzerland, April 21, to 23 as a result of the American action.

Lord Killanin, president of the International Olympic Committee, said the meeting was the result of requests by the Moscow Olympic Organizing Committee and "certain national Olympic committees" that wanted talks with the executive council.

Many national Olympic committees that have heretofore opposed a boycott are expected to change their position nowthat the USOC has made a firm decision to stay away from Moscow.

"I think there is a great possibility that many of the Western European countries and Canada, among others, will not follow our lead," USOC executive director F. Don Miller said today.

West German Chancellor Helmut Schmidt said Saturday that the Soviet Union's continuing presence in Afghanistan would make his country's participation in the Moscow Games impossible.

The West German Olympic Committee will vote on the matter May 15. If the West Germans decide to join the boycott, most of the rest of Western Europe, Japan, and a number of Third World nations seem certain to do the same, U.S. officials say.

"I think it's a very good bet that our allies willjoin us, our principal ones, in boycotting the Olympics. I think we'll see that happening over the course of the next month, and I think the whole process is greatly strengthened by what happened in Colorado Springs yesterday," Deputy Secretary of State Warren M. Christopher said in Washington today.

Cutler said the administration has no plans to launch a big, new effort to persuade other countries to join the boycott, but will continue with what he called "a major diplomatic effort, within the limits of how you talk to and deal with your own allies."

"We will continue those efforts, we will point to the decision of the USOC, and we have to do what we can to counter a very strong counter-effort being made by the Soviet Union [which is trying] to persuade -- or if necessary, to threaten to compel -- other nations to send teams to the Games," said Cutler. He reiterated administration estimates that about 50 nations will ultimately take part in the boycott.

Said Christopher, "My own feeling is that the Olympics, if they do ever take place, will only be a shadow of what was expected before the brutal invasion of Afghanistan."

Douglas F. Roby, 82, of Ypsilanti, Mich., one of two American members of the International Olympic Committee, said today that a widespread boycott could lead to cancellation of the Moscow Games. However, other sources close to the IOC said this was highly unlikely.

USOC President Robert J. Kane said he was "almost dead certain" the Games would go on, regardless of how few countries participate. Miller agreed, pointing out that the IOC has contractural obligations to the Moscow Olympic Organizing Committee.

The USOC voted today to withdraw its financial support of approximately 60 American referrees, judges and "techinical staff" who were scheduled to work at the Moscow Games.

The historic House of Delegates meeting adjourned, with many delegates expressing mixed feelings at the decision not to accept the Olympics invitation -- sadness at the sacrifice they forced on American athletes, patriotic pride, and an almost universal sense of relief that a decision has now been made, ending three months of uncertainty.

"The pressure has been great, but I am confident we did what was right under the conditions," said Kane.

The man most responsible for swaying undecided delegates to vote for a boycott, nearly everyone agreed, was Simon, a 16-year member of the USOC and its current treasurer.

When the outcome of Saturday's meeting seemed most in doubt, the former treasurey secretary and energy official in the Nixon and Ford administrations delivered a 20-minute speech urging the delegates to support President Carter because it was their patriotic duty and a matter of self-preservation for the USOC.

Simon's speech, described by delegates as "stirring" and "extremely persuasive" brought a lengthy standing ovation. Shortly thereafter, the delegates voted not to send a team to Moscow.

Simon had arisen at 4 a.m. Saturday morning to draft his remarks, which systematically addressed most of the arguments made on behalf of going to the Games.

He advised delegates to disregard charges that Carter's call for a boycott was politically motivated. "This is a moral choice we have to make. When the president of the United States makes a determination on national security, there is no way he can be denied," Said Simon.

Simon also said that a vote to defy the president would have "put the USOC right out of business." He made this point emphatically in his speech, telling delegates: "The USOC depends critically on the support of the American people for our very existence. We cannot afford to be perceived by the American people as placing our own selfish interest ahead of the national interest."