Vice President Mondale launched an effort to extend the Carter administration's human - right campaign into southern Africa today, holding six hours of talks with South African Prime Minister John Vorster.

Both sides refused any comment on the meeting as they left the Hofburg Imperial Palace in the center of the Austrian capital.

In their public statements going into the meeting - seen by both sides as crucial factor in resolving guerrilla wars in Rhodesia and Namibia (Southwest Africa) and in the future of white rule in South Africa - the two men apepared to promise an unyielding exchange of conflicting views.

They announced unexpectedly, however, that they were adding a second hour of talks Friday morning, indicating that an immediate impasse had not developed.Mondale is to leave for Belgrade Friday afternoon.

South African officials fear that the Carter administration's 'Africa for the Africans' policy of the early 1960s.

South Africa's new foreign minister, R. F. (Pik) Botha, set in motion the process that led to today's meeting, in large part to let Vorster explore this troubling prospect firsthand.

Henry A. Kissinger met Vorster three times last year and implicitly relaxed American pressures on South Africa in return settle the Rhodesian conflict.

Mondale's guarded statements to reporters since he left Washington Saturday suggest that he came to Vienna to stress to Vorster that the Carter administration, while eager to get his help on Rhodesia, will not tolerate apartheid in return for Vorster's assistance.

He was also expected to emphasize America's own experience in lessening racial violence and hatred as a suggestion of the direction the United States would like to see South Africa take.

Seated beside Mondale was Ambassador Donald F. McHenry, a black diplomat who is the State Department's top expert on Namibia and who engaged in negotiations with Vorster on the future of Namivia earlier this month in Cape Town.

"I believe you have taken a trip since we last met, a holiday in Tanzania," Vorster twitted McHenry before the meeting formally began. McHenry smiled at Vorster reference to several days McHenry spent in Dar es Salaam waiting in vain to meet a Namibian exile leader.

Mondale and Vorster met alone for 50 minutes during the 2 3/4 - hour morning session. Mondale then telephoned President Carter, as he did at the end of the afternoon session.

Vorster has indicated that his concept of the meeting differs sharply with that of the American side. He has apparently come to try to persuade Mondale that the West cannot afford to cripple a white - minority government that protects a billion dollars of American investment, guards shipping routes to the Middle East and is increasingly threatened by the expanding Soviet presence in Africa.

Mondale told reporters yesterday that while the meeting was a result of 'a parallel initiative' by both countries, had personally relayed to President Carter in March 'a request from Prime Minister Vorster for anemissary to discuss our relations.'