The resignation last week of the conservative chairman of the Broederbond secret society and his replacement by a reformist constitute Prime Minister Pieter W. Botha's most important victory so far in the struggle for control of the Afrikaner nationalist movement that has ruled South Africa for 35 years.

But in what could prove a pyrrhic victory, observers here expect a purge of anti-Botha members, which could seriously reduce the potency of what is generally regarded as the most important organization in the Afrikaner power structure.

Such a purge took place after a small group of far-rightists split from the ruling National Party in 1969. Last year a much bigger group broke away under Andries P. Treurnicht, himself a former chairman of the Broederbond, to form the Conservative Party.

A leading Afrikaans-language newspaper, Beeld, estimated last week that 60 percent of Broederbond members favor Botha's plans for token constitutional reform of the country's segregationist system, apartheid.

That suggests that 40 percent are against him. "Botha could end up with a broken reed in his hand," one political commentator has warned.

The Broederbond was founded in 1918 to advance Afrikaner commercial and language interests, and to map a strategy for the Dutch- descended Afrikaners to gain political dominance and impose apartheid. The Afrikaners make up 60 percent of South Africa's white minority.

Their goal was achieved in 1948. Since then, the Broederbond's importance has diminished, but it still has an almost mystical reputation as a cornerstone of Afrikaner nationalism.

Membership is by invitation only, and members take an oath of secrecy at an elaborate, semireligious initiation ceremony. An executive committee called the "12 Apostles," presided over by a chairman, runs the society.

The membership itself is kept secret, but it is widely believed that all important Afrikaners are members. That includes all Cabinet ministers, senior civil servants and leaders of the country's three branches of the Dutch Reformed Church.

In recent years Afrikaner prime ministers increasingly have used the Broederbond both as a think tank to evolve policies and as a network for getting policy ideas accepted by key community leaders.

After Botha announced his constitutional reform plan granting limited political powers to mixed-race "Coloreds" and Asians, although not to the black majority, it was evident that he would come into conflict with the Broederbond. It was then under the chairmanship of Carel Boshoff, head of the theology department at Pretoria University and a known conservative sympathizer.

Boshoff is a son-in-law of the late Hendrik F. Verwoerd, the assassinated former prime minister who was the chief architect of the apartheid ideology.

Although Botha's reforms are only token, archconservatives accuse him of dangerous revisionism, and Treurnicht's pitch is for Afrikaners to return to the true path of Verwoerd.

Boshoff holds other important positions, too, in the heavily institutionalized Afrikaner community. One is the chairmanship of the South African Bureau of Racial Affairs, an important think tank on race policy that invented the word apartheid in the 1930s.

Botha badly wanted the bureau to endorse his constitutional plan, which would have helped legitimize it in the eyes of Afrikaner intellectuals. But last week it published a report sharply criticizing it.

This triggered the Broederbond crisis. Angry pro-Botha members of the society confronted Boshoff at a secret meeting July 4 and, according to reports in Afrikaner newspapers, reminded him that the chairman of the Broederbond was supposed to avoid taking a politically contentious public stance that might damage Afrikaner unity.

In effect, they told Boshoff he should choose between his chairmanship of the secret society and his association with the bureau's report.

He chose the bureau, resigning as chairman of the Broederbond but remaining one of the "12 Apostles." His place as chairman was taken by Pieter de Lange, principal of Johannesburg's Rand University and a well-known proponent of the Botha reforms.

The government also cut a $70,000 annual subsidy to Boshoff's bureau, a sign of the vindictiveness this fratricidal struggle is generating.

With de Lange in charge of the Broederbond, Botha clearly will be able to use it to propagandize his cause among Afrikaners rather than have it working against him. This is a major gain for him.

But the question increasingly being asked here is how far the once invincible Afrikaner nationalist movement is being weakened by this struggle.

As the shakeout continues, one organization after another is being split. Some, like the Broederbond, are breaking Botha's way; others, like the racial affairs bureau, toward Treurnicht. All are being weakened.

The Afrikaner Students Union, began its annual conference in Pretoria this week amid speculation that it would split, with pro-Treurnicht members gaining the majority.

On the heels of that meeting, the important Federation of Afrikaner Cultural Unions began its annual conference here Tuesday. The federation is the Broederbond's cultural wing, and Botha is thought to have the edge there.

Meanwhile, Boshoff, released from his supposed position of neutrality as Broederbond chairman, is free to become more vocal. As chairman of the Voortrekkers, a youth organization similar to the Boy Scouts, he can be expected to try to win that organization over to the far-rightist cause.

Botha consequently may grow more cautious with his reformism as he loses some of the battles.

As the Johannesburg Star, South Africa's biggest daily, said in an editorial: "As the broedertwis brothers' quarrel continues, not all the victories may go to the National Party. In consequence Mr. Botha is likely to run scared, retreating even further on already overcautious changes."