The ruling National Party and rightist Conservative Party each held on to their parliamentary seats today in three fiercely fought special elections that were seen as a test for Prime Minister P.W. Botha's plan for limited reform of its racial policies.

Andries P. Treurnicht, leader of the far right Conservative Party, easily retained his parliamentary seat of Waterberg against the ruling National Party, while the National Party in turn easily retained a Pretoria seat where the integrationist Progressive Federal Party was thought to have a chance of scoring an upset victory.

In both instances the winners increased their majorities, compared with the last general election in 1981.

The National Party also retained a Transvaal Provincial Council seat in the gold-mining town of Carletonville, west of Johannesburg, where the far rightists had hoped to capitalize on discontent among white miners at recent changes in racial segregation laws allowing blacks to do more important work and to join labor unions.

Only whites were allowed to vote in the special elections.

Prime Minister Botha's proposed reforms, which were an issue in the election, would create separate legislatures elected by Asians and mixed race "Coloreds" but would continue to deny the vote to the nation's 20 million blacks. The present legislature, representing 4.5 million whites, has no representatives elected by blacks, the 900,000 Asians and 2.5 million Coloreds.

A warning sign to the government here was that the combined total polled by two far rightist parties, the Conservative Party and the Herstigte Nasionale Party, was higher than the National Party total. That means if the far right parties can form an election pact, which they tried but failed to do for these special elections, they could defeat the government in seats like Carletonville.

The Carletonville election was held to fill the vacancy left when a member of the Provincial Council died. The other two elections, and the fourth one still to be declared in the country's northernmost district of Soutpansberg, resulted from a challenge that the minister of labor, Stephanus P. (Fanie) Botha, issued to Treurnicht last February.

In a headstrong moment during a debate, Botha dared Treurnicht: "You resign your seat and I'll resign mine." He also challenged Thomas Langley, a conservative legislator from Pretoria, to be his opponent in Soutpansberg.

Treurnicht and Langley accepted the challenges.