Looking back sharply, Lord Michael Killanin, former head of the International Olympic Committee, today criticized former president Jimmy Carter for last year's boycott drive against the Moscow Summer Games, declaring that Carter and other politicians failed in their efforts and the sports world emerged as the winner.

At the same time, Sergei P. Pavlov, president of the Soviet Union's National Olympic Committee, hinted that United States support of South Africa could endanger the 1984 Olympics scheduled for Los Angeles.

The official remarks came before a congress of more than 600 delegates from international sports federations and national Olympic committees, plus athletes and coaches, meeting here in a once-every-eight-year exercise to draw up guidelines on the future of the troubled Olympics. Among the primary concerns of the congress is how to shield the Games from political interference.

The current U.S. tour by the South African rugby team, the Springboks, has drawn speculation about a 1984 Olympic boycott by the black African states and the Soviet Union. In his speech today, Pavlov stopped well short of a direct threat against the Los Angeles Games, but he warned that all powers that have contact with South Africa, and all that condone such contact, are subject to judgment by the Olympic movement.

"This behavior must be considered as a direct violation of the Olympic charter," Pavlov said.

Killanin, meantime, in an unusually critical retrospective on his eight years as IOC president, which ended at the close of the Moscow Games last July, described the U.S. boycott as a "testing time" for the world sports movement that "is regretted by all concerned."

The 69-year old Irish lord, who has been made honorary life president of the IOC, cautioned against mixing sports and politics.

"Sport, although it has its own politics, should not be used for political ends, especially when the normal governmental processes, economic, diplomatic and strategic, have not been tried."

He called Carter's boycott effort an "ill-advised, unprepared action" that sought to "sabotage" the Moscow Olympics, asserting that the Games are the "property of the IOC and not that of the Soviet Union."

"I am glad to say this failed and I believe the attempt is regretted by all concerned," Killanin said. "Despite the efforts of certain politicians to use the Games at Moscow for political expediency, I believe they in the end were the losers. The victors were the Olympic movement, supported by the international sports federations and many national Olympic committees who were completely free to participate in probably the best organized Games of the modern Olympic era.

"It is only sad," Killanin added, "that in some sports, competitors were forbidden to compete or prevented from competing. As always, it is the athlete who suffers when politicians meddle with sport."

Killanin was decorated today with the gold medal of the Olympic Order in a ceremony in which silver and bronze medals were given to several others who fought against pressure by their governments to avoid the Moscow Games. They included Claude Collard, president of the French National Olympic Committee; Denis Howell, Britain's sports minister; Sydney Grange, president of the Australian Olympic Association; Dawn Fraser, a former Olympic freestyle champion and current Australian swimming coach, and Anita de Frantz, captain of the U.S. women's rowing team.

Also honored was Ignati Novikov, president of the Moscow Olympics Organizing Committee.