Prime Minister Pieter W. Botha announced a major reorganization of his Cabinet today in an apparent bid to strengthen his moderate faction against a rightwing element of his party that has put a severe brake on changes he is trying to introduce.

Among the Cabinet appointees announced in Pretoria is armed forces commander Gen. Magnus Malan, who will take over the Ministry of Defense from Botha, Gerrit Viljoen, South Africa's top official in the Pretoria-run territory of Namibia, will head the Ministry of White Education. He is a former university dean.

Although both men are outsiders to party politics, they are close advisers to Botha and key architects of his attempts to make limited alterations in the political setup here.

The third new member of the Cabinet is Ambassador to Britain Dawie De Villiero, who will become minister of tourism and industry. He is to be replaced in London by Minister of Colored and Indian Affairs Marais Steyn, whose hard-line remarks earlier this year during school boycotts by mixed-race and Indian students fueled their protests.

Three other ministers were replaced by deputy ministers who are all strong supporters of Botha's policies.

The Cabinet shuffle is one of Botha's most important moves to consolidate his political power since he became prime minister in September 1978.

Whether or not it will permit him to overcome a recalcitrant right wing in his ruling National Party may depend on whether the influence of the new appointees filters down to the party's lower levels.

This was recognized by Frederick Van Zyl Slabbert, leader of the official white opposition party, when he commented that it will be interesting to see how the new ministers adjust to the rigors of caucus politics.

Viljoen's appointment is especially significant because it is seen as an attempt to counter the influence of Andries Treurnicht, the most conservative member of the Cabinet and leader of the right-wing faction in the party.

Viljoen is head of the influential secret society of Afrikaner men, the Broederbond, a post he won in an election against Treurnicht. He was sent to Namibia or Southwest Africa, last year by Botha to quell a right-wing revolt among whites there against Pretoria's moves to make the territory an independent state.

He was relatively successful in this and political observers expect that one of his chief tasks will be to deal with rightist discontent within the National Party at home, especially in Treurnicht's power base, the populous Transvaal Province.

There is strong anti-Botha feeling there among the part faithful, who will hold their annual party congress next month. This is probably one of the reasons why Botha also announced today that Treurnicht would head a new ministry that oversees the civil service. This is a crucial post because the bureaucracy heavily filled with conservative party members, has been accused of holding up changes the government seeks to implement.

Another change Botha announced was the absorption of the ministry responsible for the needs of the 750,000 Indians and 2.5 million Coloreds, or mixed race, into the Ministry of Internal Affairs. This move is in line with the thrust of Botha's policy to draw Coloreds and Indians into the mainstream of white politics at the top level, while maintaining separate political institutions for blacks along tribal lines.

So far, Botha's main political device to carry out this policy, a proposed council to discuss constitutional changes, has been rejected by all nonwhite racial groups, as well as the white opposition party, because it does not include blacks.

Nevertheless, Botha today reiterated his intention to make a success of the council and named Minister of Justice Alwyn Schlebusch its chairman, a post that also makes Schlebusch the first vice president.