America's amateur athletes, the special-interest group with the most special interest in the fate of the 1980 Summer Olympics in Moscow, are polarized about President Carter's call for a boycott unless Soviet troops are removed from Afghanistan within a month.

Some athletes, unaccustomed to being interviewed at all, much less on foreign affairs, spent Super Bowl Sunday issuing statements and meeting the press.Others said they had not had time to read the papers or watch television because they had been out training.

Al Oerter, 43, who came out of retirement in order to try for a fifth gold medal in the discus, did not see Carter on television. "I was out training, of all things," he said. "I have to admit I feel a little deflated. The wind has gone out of my sails and all those cliches."

But Oerter, who was quoted last week as saying that American athletes ought to go to Moscow and do well as a sign of strength, said, "I'm still disappointed but it's national policy now and I'm actively supporting it."

Oerter said he had been disturbed by some of the statements he had read by other athletes "about how they were going to be denied their opportunity to shine in the light and reap the benefits of the Games. Hopefully, now, we're going to be able to put aside those concerns and show some unity," he said.

Joe Bottom, 24, who won a silver medal in the 100-meter butterfly in Montreal in 1976 and holds the world record in the event, said, "I think its good that we're doing something relatively early. I agree with the idea of not supporting the Russians by boycotting or moving the Olympics."

Bottom, too, said he was disburbed by the reactions of some athletes. "I reflect back to Montreal when the American team was asked about Taiwan (the Americans voted not to boycott the games even though Taiwan was not admitted). As a whole the U.S. team was selfish and naive about the power we had.

"We had the power to have Taiwan in the Olympics and we didn't realize it and we didn't use it. we really blew it last time.

"People are saying, 'Hey, I want my chance to go to the Olympics.' " think that's a little shortsighted and slefish. You can't separate sports and politics, it just doesn't happen, when you have a country like Russia. They're mixing sports and politics, too, with their troops in Afghanistan and still trying to have the Olympics."

Cynthia (Sippy) Woodhead, 15, one of America's top women swimmers, saw the president on television, "but didn't pay much attention," she said. "I'm ignoring the whole situation . . ."

Woodhead said an Olympic gold medal was not her goal in swimming, that she just does it for fun. "All the big meets are neat," she said.

What effect would a boycott have on the Russians, she was asked.

"They'll just get more medals," she said.

Tracy Caulkins, another potential gold medal swimmer, issued a statement through her mother, who was fielding the calls for her 17-year-old daughter.

A film crew was waiting to interview Tracy in the kitchen, Mrs. Caulkins explained. And though she understood the obligation of athletes to comment, you could also understand why Tracy "just wanted to leave town."

"I would be disappointed if American athletes were prohibited from attending the Olympics, although if the president and the U.S. Olympic Committee decide a boycott is necessary, I would probably do what they recommend," Caulkins said.

"I do not feel it will have a serious impact on the Russians if the United States is the only country to boycott the Games, and the ahtletes would suffer. . .

"In response to the president's recommendation to have a permanent site for the Olympics games," Caulkins continued, "this seems more financially practical, although it is more interesting to be able to travel to different countries for competition. . . .

"I cannot believe this is happening. But I will continue my training and hope the problem can be resolved, so the Games can be held."

Pat Connolly, the coach of one of America's best runners, Evelyn Ashford, said 'Essentially we will support Carter or the government in whatever moves had to be made, but we don't feel the Olympics should be part of it"

Frankly, she added later, "I think it's really a desperate move on Carter's part. It's as if he's powerless to do anything else."

Margie Weiss, of Silver Spring, who coaches Olympics gymnastic hopefuls Sherry Mann and Jackie Cassello, said "We're pretty much in support of it. Jackie was supposed to go to a (Soviet) tournament in March but we pulled her out. The Russians just don't play by our rules. I can't see having to call Jackie's mother and telling her that Jackie was detained in Moscow for taking a picture of the Kremlin or something like that."

Weiss said that "the crisis is making my kids work harder. They feel the electricity. They're realizing the true meaning of the Olympics. They see the president directly involved in what they're doing. You can't take politics out of the Olympics . . . or out of anything. Gymnastics is politics.You wear the right leotard, you win."

Albert King and Nancy Lieberman are two of the finest men's and women's college basketball players in the country, respectively. Both are from New York; both said they had beenlooking forward to the chance to play for their country; both said they were disappointed, and both said they believed the president was doing what he thinks is best.

"I know he's trying to do the right thing," said King, Maryland's star forward. "It certainly isn't my place to decide about things like that."

King has a lucrative future in professional basketball ahead of him. Lieberman does not. "Reading in the paper that Carter wants to boycott doesn't sit well with people who have given up everything for the chance to go to Moscow," she said.

"I don't think the government speaks for the athletes. The USOC does. We don't get any subsidization from the government and for them to make a drastic statement without consulting the athletes or at least the USOC has made a lot of athletes angry."

Lieberman says the public's inability to understand why the athletes are "selfish" may "create some conflict between people an athletes around the country."

Mark and Fred Borchelt, of Arlington, are brothers and rowers. They row a straight pair for the Potomac Boat Club and could win a medal. Mark, 28, says he is being fired from his job as the top administrative assistant of the Joint Economic Congressional Committed because his superiors feel he cannot devote enough time to his job while preparing for the Olympics.

"America stands for the pursuit of happiness," he said, "and my pursuit of happiness is going to the Olympics. If my going endangers lives then I can see the boycott. But the other things Carter has and will do will be more effective than using the Olympics as a political tool. By competing, I don't think I'm being insensitive to Afghanistan."

Mark's younger brother Fred, who has been training for 12 years, said, "There's no way that by competing with in a sports framework, that we can help the Soviets. The world knows of their atrocities."

"Everyone decries the Olympics for being political," he added later, "but it's the politicians to make it that way."

Robin Campbell, another local Olympic hopeful, picked up the phone in California where she now lives and trains. She had not heard the news. The text of Carter's statement was read to her. Her silence echoed the resignation of many other competitors. "Well," she said finally, "what am I supposed to say?"