With both moderates and ultrarightist white supremacists cutting sharply into his popular support, Prime Minister Pieter W. Botha wrested a costly victory in yesterday's all-white elections that signaled widespread grass-roots discontent with his leadership.

The failure to win a straightforward endorsement for his policies of gradual change is reflected in the final result that showed Bothahs National Party losing 15 percentage points in the total popular vote compared with the last parliamentary elections in 1977.

More significantly, however, the outcome revealed the beginnings of a potentially serious right-wing backlash among the Afrikaners who make up the bulk of National Party supporters.

How Botha will respond to the twin challenge will become clearer when Parliament opens July 31.

But given the deeply ingrained desire by Afrikaner leaders for unity among their people and the possible psychological effect within the National Party of the strong right-wing showing, yesterday's outcome does not augur well for those who favor rapid dismantling of racial discrimination at home and urge bold initiatives abroad.

In his first television interview since the election, Botha tonight said he would continue to seek to "inspire the President's Council to come forward with proper proposals" on constitutional change.

The council is an appointed body set up to look into constitutional reform, but it has no black members to represent the country's largest group.

His remarks were a reformulation of his past vague statements on his plans for reforms rather than a resounding affirmation that he would not be deterred by the rebuffs he has taken from both the right and the left.

Botha's party was returned to power with a massive majority of 131 of Parliament's 165 seats, but it recieved only 53 percent of the total popular vote compared to the 68 percent it received in the last elections in 1977.

The ultrarightist white supremacist Herstigte (Reconstituted) National Party drew 191,249 votes, which was 157,000 more than in 1977 and accounted for 13 percent of the popular vote. It did not win any parliamentary seats but helped to slash the majorities of many National Party candidates.

Andries Treurnicht, a Cabinet minister and the standard-bearer of the hard-liners in the National Party, kept his seat by a slim margin of mere 1,461 votes in a hard-fought battle with the Herstigte leader Jaap Marais. In 1977 Treurnicht won his seat by a majority of 4,661 votes.

At the other end of the political spectrum, the moderate Progressive Federal Party won an additional eight seats. One of its candidates defeated Botha's minister of commerce, industry and tourism, Dawie De Villiers. It was the first time a Nationalist Cabinet member was defeated at the polls since 1948.

A major factor in the gains of the Progressive Party is thought to be the appealing and polished performance of its new leader, Afrikaner Frederick Van Zyl Slabbert.

Slabbert tonight said that Botha ought to "take hope" from the fact that the official opposition has grown because "what we stand for is the direction we hope he will move toward and that is systematic reform."

Becuase most National Party politicians believe there is a ceiling on the Progressive Party's support due to its "radical policies," and because they believe it appeals mostly to English-speaking whites, its gains yesterday are not likely to cause as much anxiety as those made by the Herstigte party, whose inroads were among Afrikaners.

This is probably what Marais had in mind when he boasted today: "Do not underestimate the panic that will break out in government circles over these results."

An indication of the tensions aroused was the fact that Botha's first reaction to the election results late last night was to criticize the Herstigte party for waging "the dirtiest campaign I have seen in my life. These people do not belong in a decent community and they must be removed from our public life."