Several members of the United States Olympic team said yesterday that they would not attend President Carter's White House dinner-reception Wednesday night. The event marks the end of the five-day "Olympic Honors Program" held to celebrate the efforts of 505 U.S. Olympic Team members denied the chance to compete in the Moscow Games by the U.S.-led boycott.

"Some of the things I would say to President Carter, if I had the chance to talk to him, would be pretty rude," said Seattle's Jan Harville, a member of the rowing team. "That's one reason why I'm not going."

"He (the president) has no conception of amateur athletics," said Peggy McCarthy, also a rower. "So I prefer not to go to the White House, either. A lot of people feel very strongly about going to the White House reception after Carter's position."

Carter requested that the American athletes boycott the Games after the Soviet Union's military invasion on Afghanistan.

It is widely rumored that some athletes have planned to demonstrate at the president's party and/or at the Wednesday afternoon gold medal presentation open to the public on the west steps of the Capitol.

"I don't see what good a demonstration or protest would do now," Harvell said. "It would be pointless now because we still can't go to Moscow. I think not going to the White House would be a protest in itself."

Harvell and McCarthy are among the more than 400 U.S. team members who have already arrived in Washington for the celebration, which includes wining, dining and sightseeing.

Several busloads of athletes descended upon Arlington's Marriott Twin Bridges Hotel yesterday afternoon, transforming it into a red, white and blue Olympic Village of sorts.

Several of the team captains held a press conference at the hotel to discuss everything from the current boycott to their views on the 1984 Summer Games in Los Angeles.

"I'm greatly appreciative that the United States Olympic Committee has sacrificed the money and time to have this honors program for us," said Bob Nieman, pentathlon captain and two-time Olympian."But most people don't realize that we probably would have wound up here going to the White House after the Olympics anyway, probably two weeks later, that's all."

Nieman is one of the few Olympic athletes who closely follows televised and printed news accounts of the Moscow Games.

"We (the U.S. team) would have claimed three of the top five (pentathlon) spots," he said. "The silver medal was a shoo-in. I probably could have placed second if I could have beaten my own teammates. We would have been vying with the Russians for the gold.

"It pains me to watch it sometimes. I get that longing to be there competing, but I still watch."

While most of the team members were assembling in Arlington, four members of the Olympic diving team put on an exhibition at Congressional Country Club for more than 500 spectators, most of them children competing in the club's diving championship.

"When I first started this exhibition I was nervous, just like I am in competiton," said 21-year-old Kevin Machemer. "As soon as I stepped up on the board I started thinking, 'Hey, this could be Moscow,' and about how I might not make it next time (1984)."

Barb Weinstein, gold medalist in the Pan American Games last year, said she was enjoying the Washngton program, but added, "Being here definitely makes me think about being in Moscow even more."