The Reagan administration is faced with growing disarray in its entire Africa policy, the result of recent congressional actions, shifting alliances among pro-Western leaders on that unsettled continent and a series of actions by South Africa.

The administration toughened its rhetoric toward South Africa on Friday by calling upon the government there to lift its state of emergency. But the action appeared to underscore the lack of success of the administration's policy of "constructive engagement," or quiet diplomacy, in dealing with that racially troubled white-ruled nation since 1981.

Less noticed are a series of setbacks to other elements of the administration's overall Africa policy that have left officials looking for ways to cope with the mounting economic and political instability on the continent.

The policy right now, remarked one congressional aide, "looks like a piece of jumbled crochet work."

Despite the jumble, administration officials from President Reagan on down insist that no full-scale review of its Africa policy is under way and that Washington intends to stick to its much-criticized constructive engagement strategy in trying to nudge the stubborn whites of South Africa toward scrapping their racial segregation policies.

"We're not undertaking a basic policy review," remarked a White House official. "We're looking at the situation very closely to try to determine the best course to get the South Africans back on what we consider the best course."

The administration's focus right now, he said, is on examining ways of using the "very important" relationship Washington has with South Africa to bring to bear its influence more effectively. "The United States is the one country South Africa listens to at all," the official said. "Our influence is there and can be exercised at various levels."

There is little sense of urgency among administration officials, however.

Assistant Secretary of State Chester A. Crocker, chief architect of the administration's constructive-engagement policy, is vacationing in Europe and not expected back before early August.

The U.S. ambassador to South Africa, Herman W. Nickel, who was called home for consultations on June 15, has also left on vacation.

Neither has been called back to deal with the worsening crisis.

The most dramatic diplomatic step the administration is mulling over is whether to hold a high-level meeting with South Africa to dramatize its concern. Secretary of State George P. Shultz and other top U.S. officials have hinted at this in the past few days, but administration spokesmen continued to insist through Friday that no decision has been reached.

Other steps said to be under consideration within the administration include:

*Keeping Ambassador Nickel here for the time being as a sign of displeasure, or appointing a replacement with a mandate to speak out more forcefully. The latter step was proposed earlier by Sen. Nancy L. Kassebaum (R-Kan.), chairman of the Senate Africa subcommittee.

*Refusing temporarily to accept the credentials of the new South African ambassador, Herbert Beukes, until Pretoria lifts the state of emergency or takes other measures to improve the situation.

*Reducing the number of South African consulates allowed to operate in the United States or possibly withdrawing other U.S. diplomats stationed in South Africa as a further sign of protest.

*Adopting and announcing some of the measures passed by the Senate in its South African economic sanctions bill. These include increased economic aid for the black population, no new bank loans for the South African government and insisting that U.S. firms operating there observe the so-called Sullivan principles that regulate standards for dealing with nonwhite employes.

Republican congressional sources say the last step could become even more attractive as an option if no agreement is reached between House and Senate conferees next week, or if action is deferred until September, on their differing bills imposing economic sanctions on South Africa.

The troubles for the administration's Africa policy are not limited to South Africa itself, by any means. Congressional and other critics of U.S. policy point specifically to the following upsets during the past 10 months:

*U.S. efforts to isolate diplomatically Libyan leader Muammar Qaddafi, regarded by the administration as one of the world's foremost supporters of international terrorism and an arch foe, have been stymied by his ability to reach agreements with two key U.S. allies -- Morocco and Sudan -- and the refusal of our European allies to go along with Washington's "containment policy."

*Negotiations that have been under way for more than four years to negotiate a peaceful transition to independence for South African-administered Namibia have come to a halt.

*A carefully devised strategy to wean Marxist Mozambique away from its Soviet backers and socialist commitment has been battered by Congress, which voted down the administration's request for $3 million in military aid and prohibited any of the requested $15 million in economic support funds being spent until Mozambique sends home almost all its East bloc military advisers.

*Painstaking efforts to persuade the Marxist government in Angola to accede to the withdrawal of 30,000 Cuban troops, which were beginning to bear fruit, have been recently upset. First, South Africa attempted in May to blow up Angola's main oil-production center, and then Congress voted to lift a ban on aid to pro-Western rebels fighting there. Angola, in protest, has cut off talks on the Cuban troop withdrawal issue as well as on a Namibia settlement.

*Sudanese President Jaafar Nimeri -- a longtime close U.S. ally -- was overthrown by his military on April 6. The largest recipient of U.S. aid in black Africa, Sudan is beset by a multitude of conflicting political forces that has clouded its future relationship with the United States.

Events in Sudan since Nimeri's fall have highlighted another recent trend unfavorable to the administration, namely Qaddafi's success in breaking through the diplomatic isolation the United States has tried to impose on him.

Since April 6, the new Sudanese military leadership has restored diplomatic relations with Libya, signed a military protocol Washington has looked upon with "grave concern" and moved closer generally to Qaddafi in hopes of ending a Libyan-backed insurgency in southern Sudan.

Last September, Qaddafi signed a merger agreement with King Hassan II of Morocco, another close U.S. ally, and early this year also renewed diplomatic relations with Somalia, whose main foreign alliance is with the United States.