In one of the most important special elections here in 29 years, South African Foreign Minister R.F. "Pik" Botha registered an overwhelming victory for a seat in Parliament today in a fiercely contested race against an ultra-conservative that was viewed as a test of public attitudes toward change.

The election for a vacated seat from Westdene, a suburb outside Johannesburg has also been considered a major showdown between the two factions in Afrikanerdom, the descendents of the 17th Century Dutch settlers who have dominated the government since 198.

Botha's landslide victory was the first solid indication that the ultra-conservative Herstigte Nasionale Party (HNP), which supplied Botha's opposition, does not enjoy wide support even in a conservative district. Fear of defections to the HNP has been cited as one reason for the government's hesitation is endorsing major recial change.

Botha, who until two months ago was ambassador to the United States, ran on what was considered, in South African terms, the most moderate platform since the Afrikaner-dominated National Party took power. In one campaign speech he said: "I am prepared to go to war over our right to exist, but I'm not prepared to die for signs in a lift," referring to separate amenities for separate races.

He has declared openly that social discrimination in South Africa should be eliminated, although he still stands behind separate political power through African homelands - a plan to give blacks statehood and political sovereignty in nine tribal reserves over the next few years.

His opponent from the HNP - the most right-wing white group - took one of the most conservative lines in recent history, telling voters it was a medical fact that the African brain was 15 per cent smaller than the white brain. "Somewhere between the ape and the white race lies the black race," Jon Nel said in a recent address.

The HNP during the bitter campaign was: "With Pik Botha Your Future's moderation would lead to too many rights for South Africa's 18 million blacks, who outnumber whites by more than four to one, and endanger continued white rule. The HNP also charged that Botha was a "paid agent for foreign powers."

Although Botha was considered an easy winner, the results were watched closely for a sign of just how serious the HNP. Afrikaners, who make up the HNP threat was to the government party.

Many observers here contend that Prime Minister John Vorster's "caution" about moving toward change is attributable to his fear of about 60 per cent of the white population here, are highly sensitve about splits in the ranks of their traditionally tight-knit community, held together by political, religious social and historical ties.

Although the HNP is the smallest white party - never held a partiamentary seat since it broke from the National Party in 1969 - government officials have often expressed concern that any significant social change might lead to a mass exodus from their party over to the HNP.

Nel said recently that white voters where "fed up" with the dismantling of apartheid and South Africa's repeated humiliations abroad under the Vorster leadership - a claim that has long concerned the National Party as a serious possibility.

Botha's victory undercuts the fears of the HNP. As the liberal Johannesburg Sunday Times editorialized last week:

"Most sensible people will be hoping that the election proves conclusively that the HNP is nothing to be scared of - that it is and always will be no more than a dotty, right wing fringe with miniscule support."

Botha's victory also has important implications for the National Party, which is itself divided between two camps - the verligte (Afrikaans for enlightened) and the verkrampte (hard-core).

Since the first incidents of recent racial unrest in June 1976, there has been increasing concern among verligte nationalists about more dramatic chages to help alleviate black tension and to counter increasing world condemnation.

According to sources in the broederbond - the secret Afrikaans brotherhood - six leaders from government, academia, journalism and the business community have started doing some tough and unprecedented talking about the apparent obstinency of the Cabinet against meaningful measures of change.

The marismatic, 44-year-old foreign minister still may have a stiff assignment ahead of him if he is to live up to his election promises.