South African Prime Minister Pieter W. Botha won a significant victory over right-wing rebels in his ruling National Party today and now is better placed to move ahead with planned reforms.

The rebels, headed by the conservative leader of the party's powerful Transvaal Province branch, Andries P. Treurnicht, had threatened to take the whole branch into opposition.

Instead, Botha won a resounding 172-36 vote of confidence in an emergency session of the branch's ruling committee in Pretoria. The delegates also removed Treurnicht from the provincial leadership and suspended his party membership. Other rebels also were suspended, party spokesman F.W. de Klerk told a news conference.

The prime minister still faces the biggest split his party has suffered since it came to power 33 years ago, as well as the prospect of a new parliamentary party on his right that could make inroads into the conservative white Afrikaners, who traditionally support the National Party.

But he has contained the split, and he has rid himself--at surprisingly little cost--of Treurnicht.

Botha's victory today seems likely to be welcomed in Washington. The Reagan administration, in the view of some observers here, has gone out on a limb with its policy of "constructive engagement" with South Africa, believing Botha needs encouragement rather than public criticism to enact reforms.

This week's political crisis did not address country's main problem, the lack of civil rights for the African majority. The dispute was over a hint by Botha of increased rights for the country's 2.5 million Colored (mixed race) minority, an idea that prompted the conservative resistance. Tensions between the party's conservative and reformist wings in the past have been papered over in the interests of party unity.

The white Afrikaner community, which dominates the government, feels vulnerable because whites are outnumbered more than 4 to 1 by Africans.

The tensions erupted into a split Wednesday that began with a statement in a party pamphlet that "there can logically be only one government in the country."

Party ideology opposes any form of power sharing with nonwhites, and there are no serious differences between conservatives and reformists on the nation's 21 million Africans. Both support the policy under which blacks are considered citizens of 11 small tribal homelands, not of South Africa.

But there are no homelands for the Colored and Asian minorities, who number 3.5 million. Party policy is that there should be separate Colored and Asian houses in a three-chamber Parliament, with only the white chamber having legislative powers.

An advisory panel to Botha has been working on proposals for a new constitution expected to include a single Parliament with Colored and Indian representation. The pamphlet's reference to "one government" were taken by some of the conservatives to be a reference to this and to signify Botha's approval, a stance bolstered by his comment in Parliament that he favored "healthy power sharing."

On Wednesday a call in the parliamentary caucus for a vote of confidence in Botha resulted in 100 members voting for him, 22 against.

The split quickly developed into a struggle for the powerful Transvaal branch, which has half the party's 142 parliamentary seats.

Treurnicht had no comment after today's vote.