FOR THE ADMINISTRATION, the question of whether the United States should join the United Nations' cry for economic sanctions, or penalties, against South Africa for its latest political atrocities is an exercise in the new diplomatic math. Having linked relations with South Africa to Pretoria's domestic policies, the United States can't not react without losing credibility at home, in black Africa and in Pretoria. To overreact, however, could diminsih South Africa's necessary cooperation in the Rhodesia and Namibia crises and put its government into an even more perverse adn embattled mood. The administration must also avoid taking a stance so far in front of general public opinion - which remains, we believe, ambivalent - that Pretoria will be able to exploit the gap.

We don't like sanctions. This has nothing to do with favoring apartheid. It has to do with the conviction that sanctions are a poor and possibly self-defeating tool to use against a system so powerfully entrenched. Only if sanctions were intensified close to the point of a declaration of war would they likely do more than embitter and solidify most whites. How will the international community care for the vulnerable blacks inside South Africa - and also outside, in Botswana, for example, or Lesotho - who would be the first and principal victims? To start down the sanctions road is not what a responsible government ought to do just to satisfy its outrage or to keep up with the international Joneses. The effect on the people meant to be moved and helped is the first thing to keep in mind.

South Africa is no banana republic in which, if Washington chose, it could blow the system down. It resembles the Soviet Union in the sense that its white rulers are fiercely nationalistic and clannish and tend to respond by defiance to excessive outside pressure. South Africa has large stockpiles of the things that might be cut off by sanctions. It has devoted years, and its considerable wealth and ingenuity, to devising ways to render itself relatively immune. Among the ruling whites, under the manipulation of their chosen leaders, sanctions would doubtless heighten morale and the spirit of common sacrifice, rather as they have in neighboring Rhodesia, which, with a sizteenth as many whites, has survived andeven prospered under total sanctions for a dozen years. What finally brought Ian Smith to the bargaining table was not sanctions but guerrillas.

Many Americans may not wish to be told that their government, even in conjunction with other Western governments (which are even more economically dependent upon economic ties with South Africa) and with Third World governments (many of whom will continue to trade with Pretoria even as they vote sanctions), cannot itself undo apartheid. But the beginning of wisdom here is an awareness of the United States' own limitations. The feasible and effective steps the United States can take to end the system must necessarily be small compared both with the steps that the nonwhites of South Africa will take anyway and with the steps that self-interest may ultimately lead the whites to take themselves.