President Reagan prevailed yesterday in a Senate showdown vote on sanctions against the white-minority government of South Africa, but Senate Democrats moved immediately to force a second vote Wednesday on the sanctions legislation that the president opposes.

After a day of maneuvering on Capitol Hill, it was clear that Reagan's announcement yesterday morning that he was imposing key provisions of the sanctions legislation by executive order had not defused the politically volatile issue or necessarily headed off a confrontation between Congress and the administration over policy toward South Africa.

Republican and Democratic leaders in the Senate predicted a closer outcome in the second test vote.

Yesterday, the Democrats fell seven votes short of the 60 necessary to invoke cloture and force a vote on the sanctions bill, which earlier passed the House overwhelmingly.

The relative closeness of yesterday's vote reflected continuing congressional dissatisfaction with the administration's South Africa policy and skepticism that Reagan's executive order would have the same impact as would sanctions legislation.

Senate Foreign Relations Committee Chairman Richard G. Lugar (R-Ind.) said Reagan's order was "our bill in executive order form," but he acknowledged that "it is not as good as a law."

Lugar and other Senate Republican leaders said they would watch closely to make sure the executive order is vigorously enforced.

"For the moment, South Africa and the world must know that the American people speak with one voice," Lugar said.

"It seems to me we have prevailed. The president has agreed to many of the things we wanted," said Majority Leader Robert J. Dole (R-Kan.).

Dole accused Democrats of trying to "punish Ronald Reagan" rather than South Africa by insisting on a vote.

However, most congressional Democrats, and some Republicans, were sharply critical of the president, charging that the executive order was a political ploy aimed at buying time for the administration's policy of "constructive engagement."

"I think it's imperative we speak to the issue," said Sen. Edward M. Kennedy (D-Mass.). "The president's announcement today is well-intentioned, and it is a step in the right direction. But it is insufficient."

House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) said the president's sanctions were "chock-full of loopholes."

Rep. William H. Gray III (D-Pa.), a key House supporter of the sanctions legislation, said the executive order was an "ill-disguised and ill-advised attempt to circumvent an overwhelming bipartisan consensus in the Congress."

Rep. Mickey Leland (D-Tex.), the chairman of the Congressional Black Caucus, said Reagan had offered "a superficial solution" to the South Africa issue and "the American public is not going to accept it."

"You do not set policy by an 11th-hour loophole executive order," Senate Democratic whip Alan Cranston (D-Calif.) said.

Twelve Senate Republicans -- half of them facing reelection campaigns next year -- voted with the Democrats yesterday, in effect to rebuff Reagan's executive order as insufficient and to seek enactment of the tougher sanctions legislation. Their votes, plus additional possible defections by some of the seven absent Republicans, could lead to the kind of politically embarrassing defeat that the White House strenuously sought to avert by issuing the executive order a few hours before the Senate vote.

Yesterday's vote, which climaxed several days of negotiations between Senate Republican leaders and administration officials aimed at averting a direct confrontation on the sanctions issue, came on a motion to limit debate on the sanctions legislation, thereby cutting off a threatened filibuster by Sen. Jesse Helms (R-N.C.) and a handful of other conservative opponents of the bill.

Supporters of the bill said until yesterday that were confident they had the votes to limit debate and enact the sanctions legislation later in the week, sending it to the White House where it faced a veto.

The president's executive order was aimed at heading off such a showdown. The White House, arguing that the executive order accomplished the purpose of the proposed legislation, asked Senate Republicans to vote against limiting debate.

Emphasizing the importance of the issue, the administration dispatched Secretary of State George P. Shultz to Capitol Hill, where he made a personal appeal to a caucus of Senate Republicans just before yesterday's vote.

Immediately following the vote, Kennedy, a strong proponent of the sanctions legislation, introduced a second motion to limit debate on the bill, setting up the second showdown on the issue for Wednesday.

Six Democrats were absent for yesterday's vote, and Cranston predicted that all would vote Wednesday to limit debate. In that case, barring other changes in senators' positions, a defection by one of the Republicans who were absent yesterday would be enough to deal Reagan an embarrassing setback.

Criticism of the executive order yesterday was not confined to Democrats or members of Congress. Outside the South African Embassy, protesters said they would continue to press the Senate to enact the tougher sanctions legislation and to override a presidential veto.

"We are not surprised by President Reagan's speech this morning because we know he has always stood between the people of South Africa and their freedom," said Randall Robinson, a leader of one of the protest groups.

At the other end of the political spectrum, Howard Phillips, chairman of the Conservative Caucus, criticized Reagan for adopting provisions of the sanctions legislation in his executive order, charging that pressure on the white government of South Africa would lead not to a moderate black government but to the fall of South Africa to "an armed band of communist revolutionaries."