Officials of the United States Olympic Committee (USOC) met yesterday with Secretary of State Cyrus R. Vance and other administration aides, then seemed to soften somewhat their previously hard-line opposition to an American boycott of the 1980 Summer Olympics in Moscow.

They said that if President Carter requests a boycott, they will "immediately poll prospective members of the U.S. team to ascertain their feelings about such an action."

As congressional and public support continues to mount for an American withdrawal from the Moscow games in response to the Soviet invasion of Afghanistan, USOC president Robert J. Kane and executive director F. Don Miller issued a statement saying that if the administration calls for a boycott, "based on the collective view of the athletes, the USOC Executive Board will then make a decision on whether or not to enter the athletes in the games."

After a 2 1/2-hour meeting at the White House with Vance, presidential counsel Lloyd Cutler and deputy counsel Joseph Onek, Kane said that "we were informed that President Carter has made no decision on a boycott."

However, White House press secretary Jody Powell said that the president would make a decision "within a few days," and there was widespread speculation that he will propose the boycott, which he, Vice President Mondale and Vance have all said that they favor.

Vance said earlier this week that he favors an Olympic boycot if the Soviet troops are not withdrawn from Afghanistan by mid-February.

Miller said last night that "the necessity for Soviet withdrawal" as a prerequisite for American participation in the Moscow games was discussed at the White House meeting, and that his impression was that only "a de-escalation" by the Soviets would convince the administration that a boycott was not necessary.

Miller did not rule out the possibility that the USOC would defy a request by the administration for a boycott if the athletes voted against it.

"We made no statement relative to defying a presidential order or suggestion," he said. "Our attitude toward that was to say that it would have to be given extremely careful consideration in cooperation with the athletes.

"I cannot prejudge at this moment. It depends on the world situation at the time such a poll would be taken, and the reaction of our executive board," Miller added, noting that the International Olympic Committee has ruled that its affiliated national Olympic committees, and not governments, are invited to the games, and have responsibility for decisions regarding Olympic participation.

"I think the thing that must be understood is that if we are going to maintain the Olympic movement at all, for its benefit to mankind, that any decision of this nature must be made within the system," Miller said.

"This type of consideration we have before us now," he said, referring to a possible government-ordered American boycott, "is in my judgment tantamount to the destruction of the Olympic movement."

Meanwhile, the Muhammad Ali Amateur Sports Club, which includes a number of prospective Olympians, voted unanimously not to participate in the Moscow games, the first significant world-class athletes to make that decision.

"To see athletes thinking in favor [of a boycott] shows how strong the patriotic feeling is right now," said U.S. Olympic track coach Jimmy Carnes after hearing of the club's vote in Santa Monica, Calif. "More and more of them seem to be leaning that way now. I don't know if it's 50-50 yet, but there seems to be some favor for a boycott."

Congressional support for an American-led boycott unless the games can be moved to an alternate site -- an action that the International Olympic Committee again asserted was impossible -- also gained momentum yesterday.

Chairmen Frank Church (D-Idaho) of the Senate Foreign Relations Committee and Clement J. Kablocki (D-Wis.) of the House Foreign Affairs Committee both said they favored moving the games from Moscow and indicated they would give high priority to resolutions calling for relocation of the games, a boycott or organization of an alternative "Free World Olympics" when Congress reconvenes next week.

"I support moving the Olypics from Moscow. Were the game to be moved, the Soviets would not only suffer a loss of prestige but, more importantly, it would be an event the Soviets leadership could not hide from its own people, tangible evidence of the world's indignation," said Church.

"The question, however, is whether the International Olympic Committee can be convinced to seek a change and whether other countries would support such a move in sufficient numbers to make a difference."

Miller said the USOC "has been in constant communications with Lord Killanin," and yesterday received approval to present a case for moving or delaying the games to the IOC executive board.

He is not optimistic about changing the IOC's oft-stated position that the games cannot be moved or canceled, however, because the Soviet Union technically has not broken any IOC rules. "we will just have to make our very best efforts to make them understand how we look upon the Soviet Union's blatant aggressions, and what they mean to the rest of the world," said Miller.

IOC president Lord Killanin reiterated in a radio interview in Paris that his organization would not break its contract with the Moscow Organizing Committee. Asked if he were optimistic that the games would be held in Moscow as planned, Killaninsaid wryly, "If I wasn't optimistic, I would have committed suicide long ago."

But on Capitol Hill, even though Congress remained in recess, the movement to press the IOC to shift the games or risk a U.S. withdrawal appeared to be growing.

Zablocki said that the Olympic issue, which has raised a surprising groundswell of public support for a U.S. boycott, will be given "a high order of business" when Congress returns next week, and that his committee, which will receive a number of resolutions, will in some fashion "express its feelings" about U.S. participation.

"I lean toward the idea of the United States not attending the summer Olympics," Zablocki said. "We should consider this action. The Soviets should understand how seriously we view this. It would be another expression on the part of the United States about the Soviet aggression against Afghanistan."

Resolutions urging various sanctions against the Moscow games have been introduced by Reps. Robert E. Bauman (R-Md.); Don Ritter (R-Pa.) and John H. Buchanan Jr. (R-Ala.). Reps. Clarence E. Miller (R-Ohio) and Thomas A. Luken (D-Ohio) are enlisting support for similar resolutions they plan to introduce next week.

In the Senate, David Pryor (D-Ark.) has lined up a dozen cosponsors for a sense-of-the-Senate resolution urging the USOC to petition the IOC to have the games moved from the Soviet Union.

After yesterday's White House meeting, the USOC said in a statement that "the current situation makes us conclude that the Olympic movement worldwide may have to reconsider" its charter stipulation that national Olympic committees function independently of their governments.

The USOC executive board of 86 voting members will discuss this, as well as possibilities for pressuring the IOC into changing the site of the summer games, at meetings in Colorado Springs Friday through next Sunday. The IOC will meet at Lake Placid, N.Y., Feb. 10-12, just before President Carter opens the winter games there Feb. 13.

The USOC's Miller said yesterday that a U.S. decision to boycott the summer games could have a potentially devastating effect on the winter games, and ultimately on the 1984 summer games in Los Angeles. But a spokesman for the Lake Placid Olympic Organizing Committee said, "As far as hurting our games, I don't see how it could."

The spokesman, Ed. Lewi, said that the registration deadline for athletes was Dec. 19, and any who withdrew now for nonmedical reasons would face disciplinary action by the IOC.