Senate Republican leaders bluntly warned President Reagan yesterday that Congress is on the verge of approving legislation to impose sanctions against South Africa unless he moves forcefully to pressure Pretoria to begin negotiations aimed at ending apartheid.

The warnings came as White House aides scrambled to rework Reagan's planned address on the subject today, and as the president's intended nominee as the first black U.S. ambassador to South Africa, Robert J. Brown, withdrew his name from consideration.

Senate Foreign Relations Committee Chairman Richard G. Lugar (R-Ind.) urged Reagan in an Oval Office meeting never to use the term "constructive engagement" again, a Lugar aide said. The term refers to Reagan's embattled policy of seeking to end South Africa's policies of racial segregation through diplomatic persuasion rather than sanctions.

Senate Majority Leader Robert J. Dole (R-Kan.) predicted that if Reagan did not come up with a "credible new initiative" on South Africa, efforts may be made in the Senate to take up sanctions legislation this week. "We want to avoid that," Dole said.

The House has passed a tough sanctions bill, and the Senate this week has scheduled hearings on sanctions proposals. Those in favor of a major change in U.S. policy in South Africa are threatening to bypass the more drawn-out legislative hearing process and try to attach sanctions to key legislation this week.

The withdrawal of Brown, a High Point, N.C., businessman, was a major embarrassment for the president, who had planned to announce the nomination in his address today. White House aides feared his nomination would face problems in the Senate.

Officials said Reagan may still name a black ambassador to Pretoria, and a leading candidate was identified as Terrance Todman, U.S. ambassador to Denmark. Another possibility under serious consideration would be to name a special envoy to South Africa to help foster negotiations, officials said.

Reagan has repeatedly refused to impose "punitive" economic sanctions against South Africa on grounds they would hurt blacks more than help them. White House officials said his speech today would reiterate this point and broadly defend the administration's approach to South Africa, saying that efforts must be made not to drive that country away from the West.

The president, in his speech, also plans to announce that he will extend the limited sanctions imposed last year when they expire Sept. 9, White House officials said. Reagan also plans to renew the administration's call for release of Nelson Mandela, jailed leader of the African National Congress, and to suggest that the United States will make a stronger effort to reach out to black opposition leaders.

Officials said substantial revisions were being made in the speech late yesterday to respond to the concerns expressed by Dole, Lugar and Sen. Nancy Landon Kassebaum (R-Kan.), sponsor of legislation that would impose escalating sanctions.

According to a spokesman, Lugar strongly urged Reagan not to come out against sanctions in his address, giving Senate leaders some flexibility in dealing with the issue over the next few weeks. Lugar also suggested that Reagan work closely with the Commonwealth nations and the European Community in taking any actions.

Officials said Lugar also urged Reagan to bring black opposition leaders to the United States, including some leaders of the outlawed African National Congress, to dramatize the urgency of beginning negotiations.

Lugar told reporters after the meeting that he also raised the idea of sanctions that would be aimed at the white minority leadership of South Africa, such as restricting visas for them, while offering visas to blacks.

Lugar said Reagan "has to address the fact that the South African government really hasn't moved very much in the last year . . . . "

On Capitol Hill yesterday, Democrats attacked Reagan's policy during an informal hearing sponsored by Sens. Edward M. Kennedy (D-Mass.) and Lowell P. Weicker Jr. (R-Conn.). The chief witnesses were former Australian prime minister Malcolm Fraser and former Nigerian head of state Gen. Olesegun Obasanjo, cochairmen of the seven-member "Eminent Persons Group" that was appointed by Commonwealth nations last October to explore prospects for negotiations leading to democratic change.

Fraser and Obasanjo warned that failure by the United States and Britain to impose economic sanctions in the next few months would almost certainly lead black leaders to embark on a bloody, long-term guerrilla war against the South African government.

Staff writer Edward Walsh contributed to this report.