Secretary of State Cyrus Vance arrived last night for crucial talks between South Africa and the five Western members of the U.N. Security Council in an all-out effort to arrange a peaceful transition to independence for the arid but potentially rich territory of Namibia (Southwest Africa).

The talk will provide a major test for the credibility of Washington's Africa policies, in the view of observers here, and go a long way toward determining whether South Africa will face more isolation from the West and increased confrontation will Sovietbacked black guerrillas.

For both countries, the two-day meeting will define more sharply their relationship, which is likely to be one of increased estrangement since prospects for success, as one participant put it, are "dreary."

The United States, France, Canada, Britain and West Germany will be asking South Africa to allow the United Nations to oversee the move to independence for Namibia, now ruled by South Africa.

The Britain, Canadian and West German foreign ministers spent yesterday in the Namibian capital of Windhock sounding out local opinion. They are scheduled to move on to Pretoria today, joining Vance and French Deputy Frreign Minister Oliver Stirn in discussions with South African Prime Minister Pieter Botha and his foreign minister. Pik Botha, that all sides expect to be tough.

South Africa agreed last April to allow the United Nations to run the transition under a plan drawn up by the five Western powers. Last month, however, Pretoria backed out because it objected to the size of the U.N. peacekeeping force that would be sent to the territory during the transition.

Instead, South Africa said it would hold its own electrions in early December. The move is generally regarded as an attempt to prevent the black nationalist movement, the South West Africa People's Organization, which is waging a Soviet-backed guerrilla war against the South African army, from participating in the elections.

Unless the Western powers get assurance that South Africa will permit U.N.-supervised elections, they will be under strong pressure to support calls for economic sanctions when the Security Council next meets on the problem Oct. 23.

The Western diplomats are likely to stree that any unilateral move will probably result in increased involvement of Cubans, East Germans and Soviets on the side of SWAPO against the South African troops. They undoubtedly will hint that the West would find it difficult if not impossible to come to South Africa's aid in such a case.

There are no signs that the South African government is going to reverse last month's decision. Private conversations with government and military officials time and again reveal a determination that SWAPO, which is regarded as a Marxist movement, should not assume power in the vast mineral-rich territory.

A SWAPO victory would present South Africa with the loss of its last buffer against hostile black-ruled states that can offer sanctuary to black South African guerrilla groups. South Africa appears to have decided that the risk of a Vietnam-style war is preferable to such an eventuality.

The South Africans also have apparently decided that now is the time to see if the West if bluffing in its implied threat over the past 13 months to back sanctions against the white minority government if it does not cooperate with the U.N.-sponosored settlement.

The South African Cabinet appears to calculate that the economic interests of the five Western powers will keep them from supporting full trade and economic sanctions against the beleaguered country and that, in any case, effective enforcement of such sanctions would be impossible.

According to some reports, South African government agencies have been-advised that a campaign to prepare the public for the possiblity of sanctions may be necessary. Foreign Minister Botha went on television several nights ago to say it sanctions should be imposed, "we can, we will and we must survive."

A South African refused to go along with a U.N. presence in Namibia would be a sharp blow to American policy in Africa, which has aimed at limiting Soviet involvement in the arranging negotiated settlements.

South African intransigence could pose Washington and its allies with the necessity of demonstrating to black Africa that its threat to back sanctions is not empty.

Representatives of the five powers are keeping silent on what kind of sanctions they might back. According to one source they would be designed for a "psychological impact to underscore South Africa's growing isolation." Such moves might include downgrading embassies in Pretoria, prohibiting fresh investment by business or supporting an oil embargo.