In the four months since he took office as prime minister of South Africa, Pieter W. Botha has defied his image as a tough and inflexible leader, to the surprise of Western observers.

The apprehension voiced by Western diplomats at the accession of the hawkish former defense minister has given way to guarded optimism that he will prove to be cooperative with Western initiatives to find solutions for the regional conflicts in southern Africa. They are pleased by South Africa's apparent commitment ot a United Nations' plan for peaceful transition to majority rule in Namibia, a territory administered by Pretoria as Southwest Africa.

Although Pretoria's cooperation is due in large part to a more conciliatory style of diplomacy by the Carter administration, and to some changes in the U.N. plan, it is believed that Botha's news perspective from the prime minister's office has contributed as well.

"He's mellowed in office," said one Western diplomat.

There are also signs that Botha, described by many as a practical and decisive leader, may move the all-white government out of the rudderless drift it was experiencing under his predecessor John Vorster toward some preliminary solutions for the domestic racial problems tearing at South Africa.

He gave his first clue on how he intends to cope with the country's most serious challenge -- the growing aspirations of the urban black population -- when he appointed Piet Koornhof, a well-known moderate, as minister of black affairs. Druing his tenure as minister of aports, Koornhof pioneered multiracial sporting events.

Botha's first tangible act in support of Koornhof's reformist attiudes was allowing his new minister to call off a planned destruction of Crossroads, the huge squatter camp in Cape Town. Koornhof said that, with Botha's backing, he plans to relocate the 22,000 black residents of the camp without using bulldozers to wreck their homes.

In another move that could prove significant, Botha told leaders of the black homelands, the reserves set aside for this country's 18 million blacks, that his government would review the legislation setting the boundaries of the homelands. The 1936 Land Act is bitterly resented by homeland leaders because it gives them only 13 percent of South Africa, excluding all its major urban and industrial areas. Until now, the white government has refused to seriously consider its revision.

The homeland leaders, including Zulu Chief Gatsha Buthelezi, praised Botha's coureous and flexible attitude, which they contrasted with that of Vorster, but cautioned they would wait to see if Botha kept his promise.

Revision of the Land Act, however, does not address the overriding political problem of South Africa, which is the rejection of the homeland system by most blacks and their demands for sharing political power with the whites.Botha has already rejected calls for a national convention which most blacks support as the way to work out a new plitical system of shared power.

There are signs that Botha, popularly known by the Afrikaans pronunciation of his first two initials, "Pee Vee", may be increasingly diverted from governing by a developing power struggle within the National Party and by a loss of faith in their leaders among many of the country's 2.6 million Afrikaners. Both phenomena are results of the still-unfolding political scandal Botha inherited from the Vorster administration.

Many political commentators point out that Botha's handling of the scandal has done little to restore faith in the government. His well-known and feared rashness surfaced in the initial stages when he dismissed the judge who disclosed details of government funds intended for use to improve South Africa's image abroad. Then, his threats against the press for pursuing the scandal displeased the Afrikaner establishment.

Now, many Afrikaners are questioning whether a report on the scandal written by a hand-picked Botha committee told the whole truth. Suspicions were heightened by the revelation that the report's text had a mysterious gap oftwo lines and by the committee's refusal to release all the evidence it heard.

In addition, the Botha government took steps last week to silence two of the men most heavily implicated in the scandal by the report, former Information Minister Cornelius Mulder and the former intelligence chief, Hendrik Van Den Bergn.

Mulder resigned his seat in Parliament, making it clear he did so only under heavy pressure from Botha. In another controversial move, the attorney general announced he would not prosecute Van Den Bergh for his public criticism of the report, which could be taken as contempt of the investigating committee.Both actions are being seen as attempts to keep the two men from telling their sides of the story -- Mulder in Parliament and Van Den Bergh in court.

"The National Party doesn't understand how deeply their own followers are disturbed by this," said an Afrikaner editor. "It's not the original sin they mind so much; it's what is coming afterwards."

Another by-product of the scandal involves the taking of the second most powerful position in the National Party hierarchy by one of its most conservative men, Deputy Minister of Black Education Andries Treurnicht. Reportedly harboring prime ministerial ambitions himself, Treurnicht stepped into the post vacated by Mulder, who resigned because of the scandal.

Some political observers fear that Botha may not have the political finesse and acumen to come out on top in the power struggle now building up between the Botha and Treurnicht wings in the party. They point to Botha's unsuccessful attempts to block Treurnicht's election as Transvaal provincial party leader. The prime minister now finds himself in the embarrassing position of having to find a place for Treurnicht in his Cabinet according to party tradition.

The conflict is expected to sharpen when a proposed constitutional plan designed by Botha to set up co-equal parliaments for whites, persons of mixed race and Indians is discussed in the party caucus and presented to Parliament in the first half of this year. Botha is said to favor an eventual sharing of substantial power with the Coloreds (people of mixed race) and Indians. The Treurnicht faction may balk at this.