The Association of National Olympic Committees today unanimously rejected President Carter's call for an Olympic boycott and opposed efforts to change the site of the Summer Olympic Games in Moscow.

The unanimous vote came after a two-day meeting here of the association's 20-member council, which represents the Olympic committee of 141 nations.

The association, which includes some of the world's top sports managers, acts as an advisory body and normally concerns itself with setting competition standards. It is distinct from the International Olympic Committee, although membership frequently overlaps.

In a statement, the council today said that the Olympic Games "should be used only as an instrument for the furtherance of the Olympic movement and sport itself" and it appealed to the International Olympic Committee "to reject and avoid any external influence on the Olympic movement . . . designed to change existing arrangements."

A representative of President Carter, Robert Berensen of the State Department, had lobbied backstage over the weekend.

The United States, which currently has no vote on the council and is represented by other members of the Americas, had an observer here, Phillip Krumm, current president of the U.S. Olympic Committee.

Mexican President Jose Lopez Portillo hosted the association's delegates at a reception Monday. He told them they should "save the games from the political forces that are threatening them."