A bold and unexpected approach to South Africa's troubled race relations is threatening to bring Prime Minister Pieter W. Botha more political trouble than the country's worst governmental scandal, which plagued the first six months of his administration.

The trouble is coming from the right wing of Botha's ruling National Party and white ultraconservative groups increasingly unhappy with his conciliatory, mildly reformist response to the demands of the country's black majority.

It is a policy that is putting Botha in double jeopardy. Already, observers here say, it has raised expectations among blacks that, if unfulfilled, could result in worse relations between the races than before.

Botha has launched a massive public relations effort to promote better race relations and make reform fashionable, but he has been vague on details of specific changes he intends to make.

So far, he has given his minister for black affairs, Piet Koornhof, support for attempts to bridge the gulf of alienation between black leaders and the government. Botha also has initiated some reform in the country's labor laws to give, for example, legal status to black unions. Government officials have indicated privately that there are more piecemeal reforms of this nature to come.

Since Botha initiated the policy after succeeding John Vorster as prime minister a year ago, Botha has steadfastly refused to back down under growing pressure from the right wing.

Because of this stand some observers feel, Botha may witness the emergence of the ultraconservatives as a political force rather than the mere nuisance they have been until now. Many government supporters already have registered their displeasure with Botha's policy by staying away from the polls in three recent by-elections, while others gave their votes to the ultraconservative Herstigte (purified) National Party. The party has no seats in Parliament now, but appears to be gaining support.

The right-wing backlash indicates just how difficult meaningful change is going to be in South Africa, where for the last 30 years whites have been brought up on the National Party's propaganda that the system of racial separation called apartheid was the solution to this country's race problem.

Meanwhile, some black leaders are cautiously giving Botha credit for his new approach. But they remain largely unenthusiastic about his program because it falls short of offering a lasting solution to South Africa's core political problem -- the whites' resistance to black demands for majority rule.

So far, Botha's campaign has centered on jolting whites out of their complacency and convincing them that they must change if they are to survive in a country where they are outnumbered by blacks 5 to 1.

"We must adapt or die," Botha told startled followers at a recent rally. He became the first white leader to warn publicly that "revolution is not a remote possibility" in South Africa.

"It can only be averted if the government looks at the interest of all population groups and not just the whites," he added. "We can go into the future only with a message of love rather than of hate," he told party followers.

That was an unusual message from a man who, when he succeeded Vorster last Sept. 28, brought to the job a hawkish image as minister of defense, a post he still holds.

To emphasize his plea for more understanding between black and white, Botha recently toured the eight homelands or reserves that the government has set aside for blacks and the sullen all-black township of Soweto, where clashes three years ago between demonstrating students and police left hundreds dead.

As a test of the Botha administration's sincerity, blacks will be closely watching how it reacts to a move by Soweto's popular leader Nthato Motlana. The black physician this week laid a formal charge of defamation against a white mining union chief who told a newspaper that "blacks were like baboons."

The incident illustrates how Botha's new approach has put him in a vise. If authorities prosecute the right-wing labor leader, Arrie Paulus, who already has criticized the government for dropping restrictions that kept blacks out of certain jobs, Botha may face a crescendo of criticism from the right. If Paulus is not prosecuted, blacks will wonder what Botha's talk of love and reconciliation is all about.

Behind Botha's reformist direction lie some key differences in perception between himself and Vorster.

"Vorster scoffed at the idea of revolution in South Africa," said one white reporter. "Even after [the disturbances in] Soweto, he said it was an isolated incident."

Besides recognizing the real danger of black discontent at home, Botha has realized that, as he put it, "there is a relationship between South Frica's domestic policies and its international strategic option." He warned that wars and revolution would "put our security forces before an impossible task."

If Vorster ever realized this link, he never acted on it. Instead, he forestalled internal reform and attempted to improve South Africa's standing abroad by a detente with chosen black African states and by a covert multimillion-dollar propaganda "Muldergate" scandal and Vorster's downfall in disgrace.

Botha not only has steered his administration intact through the scandal, displaying more political skill than expected, but he also has shifted his government's attention fron contacts abroad to improving the situation at home.

In a slap at Voster's pandering to the right, Botha recently said, "I will lead from the front, not from the back." He added that those who do not want to come along could remain behind. He reportedly has told party insiders that he does not care if the National Party loses some seats to its ultraconservative rival party, apparently hoping he can compensate for the loss by picking up support from English-speaking voters and by gaining the cooperation of some blacks in his plans for the future.

"It points to a shift away from the politics of the tribe to the politics of survival," said newly elected opposition leader Frederick Van Zyl Slabbert of the Progressive Federal Party. "The National Party is no longer prepared to move ahead at the pace of the slowest Afrikaner. Its primary concern is no longer to maintain Africkaner identity, but to make the transition necessary for survival," Slabbert said.

Political analysts note that his policy resembles the process initiated by the white minority in Zimbabwe-Rhodesia, in which the white minority government seeks black allies from the most conservative element of the population while resisting demands for fundamental change from the militant section.

Many blacks warn that this will not be enough.

"If he is talking seriously about stopping the revolution, then he must talk about equal citizenship and full political participation for blacks and fundamental economic and social change," said Allan Boesack, a colored (mixed race) minister and black consciousness advocate.

"He may delay the revolution by doing the things he proposes, but he wont't stop it," added a black trade union organizer.