Efforts to reform South Africa's racial policies are putting Prime Minister Pieter W. Botha in an uncomfortable bind.

Caught between the right wing of his political party and a restive black population that wants more fundamental change, Botha has been trying to steer a mildly reformist course.It has been dubbed "apartheid without tears," because it stops short of changing the basic policy that has kept the country's nonwhite population of nearly 22 million under the political control of 4.5 milllion whites.

Botha's reform program is aimed at gradually dismantling the most glaring racially discriminating laws. It stems from a growing, although publicly unstated, awareness in the dominant Afrikaner community that South Africa's internationally condemned apartheid policy simply will not work in its present form.

Marking a break with the 12 years of white resistance to change under his predecessor, John Vorster, the new prime minister's program also reflects a prevailing viewpoint in the military establishment, that the "onslaught" against South Africa must be met with a double stragedy - military strength on the borders and reform at home - to defuse discontent. Botha is in his 13th year as minister of defense.

Botha's reformist moves, deliberately timed or not, can also be seen as a valuable distraction from the problems caused by the country's worst political scandal. That scandal, which involved misuse of Information Department funds, has already forced the resignation of Vorster from the largely ceremonial post of state president.

The reformist policy has been most evident in recent government moves to abolish some discriminatory labor statutes and ease restrictions on the movements of blacks. However, it has yet to eliminate a discriminatory system.

For example, a Botha administration response to a government-commissioned report on labor accepted the provision that black unions should be recognized officially and given full negotiating rights, which is a major advance. Conservative white unions, however, still will be able to determine which black unions get recognized. In addition, 2 million migrant workers, which make up the bulk of the country's labor force and much of the membership of unrecognized black unions, are forbidden from forming unions. Mixed unions are still outlawed, with exceptions.

Recently the government indicated it intends to streamline or eliminate aspects of the "pass" system that restricts where blacks may live and work. But the government is retaining other aspects even though it admitted they "create considerable human relations problems and come nowhere near completely effective control of the unlawful entry of blacks into the urban areas."

The result is that the hated pass structure, which blacks say is the most degrading aspect of apartheid, will remain largely intact. More than 272,000 blacks were arrested in 1978 for offenses under the pass system.

These changes albeit somewhat half-hearted, have been received unentusiastically in the black community. But it is perhaps the boldest venture on the reform agenda that has become the most controversial among blacks. The Botha government's main advocate of change, Black Affairs Minister Piet Koornhof, invited urban black leaders, including some outspoken foes of apartheid, jailed for their views last year, to join the government in reviewing the living standards of the 5 million to 9 million blacks who inhabit white-controlled urban areas.

Koornhof, who already has earned a measure of credit among blacks for his efforts to ammend the government's sports policy, said the proposed study signaled the government's "seriousness" about improving the lot of blacks in white areas.

"For God's sake, let's give it a chance," he said.

Up to now urban blacks have been regarded as "temporary sojourners" with no political rights in the white-controlled industrial and urban areas where they sell their labor. Since they were expected to return to the 10 "homelands" set aside for them in less than 13 percent of the country, their living conditions in the cities have always been transient and second-class.

While some radical blacks reject any dialogue with white leaders as long as apartheid remains intact, others are willing to discuss peripheral issues in the hope of enacting basic changes.

Two prominent black leaders who accepted Koornhof's invitation on those terms were black Dutch Reformed minister Sam Buti and Gibson Thula, Johannesburg representative of Inkhata, the political party of Zulu Chief Gatsha Buthelezi. They joined several black leaders who always cooperate with government initiatives.After a few initial meetings of the committee, however, Buti announced his resignation Thursday. He said he no longer believes Koornhof's effort offers a chance for meaningful changes.

Several other black leaders, notably Soweto spokesman Nthato Motlana, black editor Percy Qoboza and Anglican bishop Desmond Tutu, have declined to join Koonhof's discussion groups.

"The government is talking about change within the structure of apartheid," Motlana said. "The committees might help make the life of urban blacks more comfortable . . . But we are going to refuse to serve on any committee which makes apartheid more comfortable."

Ironically, when Motlana and Soweto's popularity supported "Committee of Ten," which he heads, asked to meet the government two years ago to discuss the same topics Koornhof now wants to talk about, they were rebuffed and Motlana was detained for five months. Now that the white government is ready to talk, it finds black political demands have become more radical.

The nature of Botha's current reformist course seems destined to perpetuate this pattern, making meaningful dialogue increasingly difficult.

Meanwhile, the prime minister has confronted a grumbling right wing, led by the minister of tourism, statistics and public works, Andries Treurnicht. These conservatives have decried the reforms as the "thin edge of the wedge" and "selling out" of white interests. They are threatening challenges to the reformist trend at upcoming party meetings.