Right-wing opposition to proposals for limited relaxation in South Africa's racial policy has presented Prime Minister Pieter W. Botha with the most serious political challenge from his own party that any South African premier has faced in the last 30 years.

The magnitude of the conservative opposition became evident earlier this month when former prime minister and president John Vorster broke nine months of silence to criticize -- indirectly but unmistakeably -- Botha's proposals on racial policy.

Vorster's reappearance on the political scene, his first since he resigned in disgrace for his part in South Africa's Information Department scandal and a subsequent coverup attempt, followed a week of "High Noon" tension between Botha and his archrival, conservative minister of public works, Andries Treurnicht.

The public squabble between the two men brought the Afrikaner-run National Party, which has enjoyed an unbroken reign since 1948, to the brink of an open split. It was the closest it has come to such a rift since a right-wing faction objecting to integrated sports broke off 11 years ago to form the Purified National Party.

Although a break was avoided last week, events have made it clear that a power struggle within the party, fueled by disagreement over the pace and nature of changes South Africa should make and by personal and political rivalries, has reached a critical stage.

Until it is resolved possibly by a split, Both's policies, his credibility and his power base within the party will hang in the balance.

Both appeared to get a shot in the arm from an opinion poll published Sunday showing that a large majority of white South Africans support his reformist policies. The poll in the Johannesburg Sunday Times showed that 85.5 percent of registered National Party members supported Botha as party leader and only 6.4 percent backed Treurnicht for the post.

Of all the voters polled, 71.4 percent said they favored Botha's racial line, 6.5 percent supported Treunicht's more conservative views and 18.5 percent supported a compromise between the two.

The right-wing challenge, which has been building up for some time, comes as the election of self-avowed Marxist Robert Mugabe in Rhodesia has further impressed Botha with the urgency for changes. In addition, international pressures to accede to a settlement in the Pretoria-run territory of Nambia are compounding this urgency. So, too, is growing violence by dissident blacks at home, exemplified by the taking of 25 white hostages in January and the recent uncovering near Johannesburg of an arms cache that included a rocket launcer.

Botha's position on changes in racial policy so far amounts mostly to words. Spurred by his generals, who urge new moves to stave off a civil war by this country's discontented and disenfranchised black majority. Botha has jolted his white followers with warnings on the need for change.

He recently told supporters that the government must take a "clinical" look at the country's problems and that Afrikaners "like all other minority groups in South Africa, would have to face a new reality."

He has suggested "changes" in the mixed marriage and immorality acts, which forbid sex across the color, bar, implying that this prohibition would be dropped. These acts "were not priorities for South Africa's survival," he said.

The 64-year-old leader also indicated that he would like to remove racial barriers in the social and economic spheres. A "states council" that would include blacks is to be set up to decide matters affecting South Africa's future.

Botha has never indicated, however, that he would take the crucial step that would resolve South Africa's political crisis -- sharing political power with blacks. Thus, his changes still fall far short of what the blacks are demanding.

But getting even these limited steps through his party caucus has been impossible because of right-wing resistance. This has raised the prospect of a confrontation with them, and a possible split, traumatic though it would be.

"The side that will have to be decided is whether reform takes precedence over unity in the National Party. The possibility of peaceful change hangs on that decision," said Frederik Van Zyl Slabbert, leader of the opposition in Parliament.

Some commentators in the Afrikaans-language press, which has backed Botha in his present course, agree with Slabbert.

"The fact is that if the future of South Africa demands it, there will be no hesitation in telling obstinate to get out of the party," editorialized Die Vaderland.

Such talk unnerves the 2.5 million Afrikaners, whites of Dutch descent who have maintained their political dominance over nearly 20 million blacks through their strong unity.

Some observers close to the National Party say Botha has been hankering for a chance to isolate Treurnicht on a minor issue and thus quell the conservatives' revolt. He got that opportunity recently when Treunicht publicly objected to the participation of a colored (mixed race) boys team in a white high school rugby tournament.

By saying coloreds should organize their own tournament, Treurnicht defied Botha's decision that government would not interfere in sporting events.

Botha who warned Treurnicht last September he would not tolerate "rebels" in his Cabinet, took the unusual step of publicly rebuking his Cabinet minister, offering what was seen as an ultimatum to either resign or back down publicly.

"The time has come for this country to realize that attitudes creating the impression that colored people are lepers must come to an end," Botha said. "I cannot associate myself with public statements or behavior which could cause further tensions between whites and colored people."

However, the rift was patched up in urgent private talks in what was considered a defeat for Botha.

"The objective was to get Treurnicht out, but he's still in -- a stone in the stomach of Botha," said one Afrikaner political analyst.

Vorster, who is said to be bitter toward Botha for his handling of the Information Department scandal, entered the fray in an address to the Afrikaner Club in the conservative town of Bloemfontein. He broadened the conflict with Botha, whom he did not mention by name, into a dispute about basic party principles by reiterating his support for the 1936 land act that apportions only 13 percent of the country's land to blacks. It was a swipe at Botha's suggestion that the act is not a "holy cow" and might be altered.

Vorster also told his white audience that he rejected the ideas of power sharing and of South African citizenship for blacks, because it would lead to the "relentless demand . . . to take them into our own Parliament."

Botha has acknowledged the difficulties of the citizenship issue for blacks, and there is speculation that he intends to introduce some kind of dual citizenship for them.

Separate development (apartheid) was the "salvation" of South Africa, and anyone who said otherwise "was not a friend of South Africa, the Afrikaner or the white man," the former prime minister declared. He added that whites should not develop a "guilty conscience" about apartheid.

At the very least, Vorster has given the conservatives a fillip in the eyes of grass-roots National Party supporters, among whom Vorster still is popular, despite the disgrace of his ouster from office.

He has reportedly been under pressure to return to public life. According to one newspaper report, at least five members of Parliament have offered to give up their seats to him. Vorster refuses to comment on the report.

His reappearance -- temporary or not -- changes the political equation for Botha.

"Vorster is a heavyweight, and in a fight between him and Botha, I wouldn't put my money on Botha," said one political observer.