South African Prime Minister John Vorster, who led his country through a period of increasing racial strife and deteriorating relations with the West, resigned yesterday, opening the way for a power struggle.

Vorster gave no reason for the resignation but he is known to be ill and there had been mounting speculation over recent weeks that he would step down. He is believed to be suffering from a circulatory ailment.

The outgoing prime minister said at a press conference that he would remain in office until his ruling National Party, dominated by conservative Afrikaners, selects his successor at a party caucus scheduled in Cape Town Sept. 28.

Two of the three leading candidates, Defense Minister Piet W. Botha and Black Affairs Minister Connie Mulder, are seen as conservatives in the same mold as Vorster, who led the party to its greatest electoral victory last November.

The other candidate, Foreign Minister Pik Botha, who was formerly the South African ambassador in Washington, is regarded as a moderate. He has little political base in the partly, however, having run for Parliament for the first time only two years ago. The two Bothas are not related.

Vorster's last major act as prime minister yesterday - announcing that South Africa would go ahead on its own without international backing to grant independence to Namibia - was seen by observers as moving South Africa further in the director of the hardliners. Thus, it was regarded in some political and diplomatic quarters as evidence that one of the two conservatives would probably succeed Vorster.

The change in leadership could bring about a realignment of moderate and conservative elements within the party, but no major changes in the country's policy of racial segregation are likely because the party is unified in its goal of maintaining aparthied.

For much of his 12 years as prime minister. Vorster managed to straddle the groups which differ mainly over the pace of change in removing some aspects of racial discrimination.

Vorster, who told the press conference that he would make himself available for the largely ceremonial post of president left vacant by the death of Nicolaas Diederichs last month, said he was leaving office with no regrets whatsoever."

Speaking first in his native Afrikaans and then in English, the prime minister thanked "all South Africans of all colors for the kindness they have shown me over the years."

At times moist-eyed, he added: "I am very thankful that God in his mercy has given me this opportunity to serve South Africa and its people in the small way in which I was privileged.

Vorster's departure brings to a close 12 years of rule by the former lawyer and tough minister of justice that saw South Africa's economic and military strength greatly expand as it enjoyed an industrial boom.

Internally, however, the country's tribe has been tested sorely by a deterioration in relations between the 4.5 million white minority and the 18.6 million blacks as Vorster and his party pushed ahead with the policy of separate development, which denies meaningful economic and political rights to blacks.

Adopting "fulfill your destiny" as his prime ministerial motto, Vorster came into power like a lion. "One of the party's young extremists," was how a newspaper described him when he became the country's seventh prime minister Sept. 13, 1966, following the assassination of Hendrik F. Verwoerd on the floor of parliament.

The aloof and unbending Verwoerd was the principal architect of separate development under which blacks are ineligible for South African political rights and must become citizens of nine "independent" ministates carved out of greater South Africa.

The pragmatic and down to earth Vorster, however, can be considered its first real implementer. Under his tutelage, the black areas of Transkei and Bophuthatswana were given independent status although no other country recognized them.

To whites in South Africa, the Vorster period has been one of great change in the status of blacks. Supporters note the inroads toward multiracial sporting facilities and competition and the relaxation of discriminatory practices in major hotels and restaurants, as well as granting of 99-year leases to black urban dwellers.

But Vorster's critics say all these changes have merely frayed the edges of separate development and skirted the basic issue which is black economic and political equality with whites. Moreover, they add, the changes have been too little, too late and lacking in commitment by the national government.

To cope with a worsening internal security situation stemming from black opposition to the government, Vorster increased police powers., curbed civil rights. These developments were a continuation of a process he began as Justice Minister from 1961 to 1966.

Although a victim himself of detention without trial for his opposition to South Africa's entry into World War II on the side of Britain, Vorster introduced legislation allowing such detention for 90 and then 180 days, as well as legislation widening the definition of sabotage.

Under his direction as justice minister, the police cracked the underground leadership of the banned black nationalist movement the African National Congress and brought Congress leader Nelson Mandela, now serving a life sentence on Robben Island, to trial.

Black anger, however, erupted in riots and demonstrations in June 1976 in Solwetto and continued sporadically for the next 16 months resulting in at least 400 deaths. The government finally carried out a massive security crackdown last October in which 49 black leaders were detained and 18 black organizations banned. In retaliation, the United Nations imposed a mandatory arms embargo against South Africa, the first such embargo against a member nation.

As opposition to South Africa's racial policies grew, the country's isolation increased as Vorster, resisted pressures for major change.

In the early 1970s, Vorster briefly embarked on a bold initiative to forge normal relations with black-ruled states to the north of South Africa as a way to counter his country's drift toward isolation. He held meetings with three African presidents, but the fruits of that detente effort were cancelled by the disastrous intervention of South African troops in the 1975-76 civil war in Angola. South Africa's relations with the rest of black Africa reached a nadir during that conflict and have not visibly improved since.