After more than four years of quiet diplomacy, time seems to have run out on the Reagan administration's policy of "constructive engagement" toward South Africa, at least in Congress.

Angered by mounting racial violence there and a lack of visible change for the better, Congress is in full revolt against this easygoing style of American diplomacy and is likely to approve economic sanctions to step up pressure on South Africa's ruling white minority to end or modify its system of rigid racial separation.

At the least, new sanctions would end all further U.S. bank loans to the South African government, sales of computers to its security forces and cooperation in the nuclear field.

All these provisions were in the bill passed 16 to 1 by the Republican-controlled Senate Foreign Relations Committee last week.

If the version passed 295 to 127 Wednesday by the House prevails, new U.S. investment there also would be prohibited, as would imports of krugerrands.

The imposition of sanctions -- considered likely because the Senate bill is sponsored by Majority Leader Robert J. Dole (R-Kan.), Foreign Relations Committee Chairman Richard G. Lugar (R-Ind.) and conservatives including Senator William V. Roth Jr. (R-Del.) and Sen. Mitch McConnell (R-Ky.) -- is strongly opposed by the administration. It would amount to a bold bid by Congress to force Reagan to adopt a new strategy toward South Africa.

In a speech in April, Secretary of State George P. Shultz argued strongly against sanctions, appealing for a "national consensus" to avoid making South Africa a divisive domestic issue. Today, he and other U.S. policy makers are faced with just such a consensus, but one they strongly oppose and seemingly have no plans to accommodate.

The White House has not given any indication whether it would veto legislation providing economic sanctions.

But a senior administration official, asking not to be quoted by name, said he saw no alternative to "constructive engagement" and strongly suggested that there would be none. If so, Congress and the administration appear to be headed for a confrontation.

If the administration sticks to its policy in the face of growing opposition and congressional initiatives, it would create a special dilemma for Republican supporters of economic sanctions. With an eye to the 1986 elections, many Republican lawmakers are eager to distance themselves from a policy increasingly viewed as far too sympathetic to white South Africa.

Since the start of the Reagan administration, the United States has sought to coax South Africa, through quiet diplomatic cajoling and occasional public reprimands, into changing. Above all, U.S. efforts have gone into trying to engineer a detente between Pretoria and its black African neighbors.

Assistant Secretary of State Chester A. Crocker, the much-criticized principal architect of this policy, was out of the country last week and making no public statements.

Crocker has angered many of his congressional critics by summarily dismissing their concern about the fate of South African blacks, including one comment he made accusing them of using the South Africa issue for purely political purposes and of exploiting "the moral equivalent of a free lunch."

Sen. Nancy Landon Kassebaum (R-Kan.), who heads the Senate Africa subcommittee and has called for the replacement of the U.S. ambassador to South Africa, Herman W. Nickel, says she believes that Crocker has not been effective in "presenting his own best case" and may have lost the ability to shape U.S. policy now. But she has stopped short of suggesting that he, too, should go.

The common refrain from Democratic liberals and Republican conservatives is that the administration has precious little to show for its four years of dealing with South Africa through quiet diplomacy. The feeling seems to be that it has spent too much time encouraging detente between South Africa and its black neighbors and far too little on pressing Pretoria to make meaningful changes in apartheid.

The administration's position is that the United States does not have enough leverage to pressure South Africa into making concessions to the black majority there. Officials say that if the United States should pull out -- not an option under the current legislation -- the Western Europeans and Japanese would be eager to buy American investments at bargain prices, and that the best we can do is to try to improve the lot of the blacks through persuasion and diplomacy.

For their part, the South Africans have made some changes in the system and declared their intention to make more. They have set up separate parliamentary chambers for persons of mixed race and Asian descent and said they would scrap laws prohibiting racial intermarriage. They also have said they will allow multiracial political parties.

The final straw for many congressional critics of "constructive engagement" came May 21 when South Africa, for reasons that remain a mystery, sent a nine-man commando team into northern Angola, apparently to blow up the oil storage tanks belonging to the partly U.S.-owned Gulf Oil Co. in Cabinda. The operation was aborted when Angolan troops intercepted the commandos, killing two and capturing a third.

The incident, according to Frank Wisner, Crocker's chief assistant, in testimony before the Senate Foreign Relations Committee Tuesday, has "clearly upset" administration efforts to bring about a peaceful settlement to a number of southern African issues. These include:

Namibia. On behalf of five Western powers, the United States has undertaken to negotiate with Pretoria the holding of elections under U.N. auspices for the independence of this territory administered by South Africa in defiance of a U.N. resolution. After four years of on-again, off-again talks by Crocker with South African authorities, negotiations are at an impasse.

Furthermore, this month Pretoria is setting up a local government in Namibia that many fear may be a first step toward a unilateral declaration of independence by the white-dominated administration there.

Angola. The administration has been seeking to persuade the Angolan government to agree to the withdrawal of 25,000 Cuban troops who went to that country to help install a Marxist government during the 1975-76 civil war. South Africa, with U.S. blessings, has insisted that the Cubans leave before it will go ahead with independence elections for Namibia.

Three months ago, Crocker proposed a phased withdrawal of the Cuban troops, but neither side has responded. Cuban President Fidel Castro, after signaling his interest in such a compromise, recently reaffirmed his determination to keep his troops there, and to send more, until South Africa gets out of Namibia.

Mozambique. In March 1984, the United States helped arrange the Komati accords, signed between South Africa and Marxist Mozambique, which were supposed to usher in detente. Each agreed to stop supporting insurgents seeking to overthrow the other's government and to respect each other's sovereignty. Since the signing, the accords have been respected more in the breach, with the anti-Mozambique opposition based in South Africa generally continuing to operate.

South Africa. The United States has been trying to encourage white leaders there to take steps to end apartheid and start a dialogue with the black majority. While the white rulers have introduced some reforms, violence there in the past six months has resulted in the deaths of more than 300 blacks.

This has created the impression that South Africa is taking two steps backward for every one forward. Crocker has tried repeatedly to persaude Congress that South Africa has crossed "a historical watershed" toward ending the apartheid system and that the changes are not "cosmetic." But he has clearly failed.