With the Carter administration's hard-won diplomatic credibility in black Africa riding on the outcome, the United States and its four main Western allies pressed South Africa yesterday to renew its agreement on a U.N. independence plan for the disputed territory of Namibia.

Secretary of State Cryus Vance and the foreign ministers of Britain, Canada and West Germany, joind by a senior diplomat from France, opened what are expected to be two days of hard bargaining with South Africa's new prime minister, P.W. Botha, and his top aides.

The first day of the extraordinary high-level mission by the five Western nations produced a surprise 45-minute private meeting between Vance and Botha, at which the secretary handed over a letter from President Carter. Vance's spokesman, Hodding Carter III, later refused to discuss the letter's contents or to characterize the day's discussions,

After a year of negotiations with an ambassadorial-level "contact group" of the five nations, South Africa agreed in April to a plan designed to bring U.N.-supervised elections and internationally recognized independence to Namibia, now known as Southwest Africa.

But Pretoria suddenly withdrew its agreement last month when prime minister John Vorster announced his resignation. The government said instead that it would push ahead with South African-organized elections to produce a local government for Namibia in December.

The meetings with Botha, a conservative generally said to have launched South Africa's deep military intervention into the Angola civil war in 1975, had been expected to give the Western nations a chance to gauge whether the South African government has adopted a new strategy of making the sparsely populated Namibian territory its front line of defense - even if it means risking economic sanctions from the United Nations and political isolation from the West.

This course is reportedly being urged on Botha by some of his military leaders, who argue that sanctions will not be enforced even if voted in the United Nations. This view holds that public opinion in the United States and the other Western countries can be swung over eventually to the South African side and used to undercut international pressure.

Such defiance would create a major dilemma for the five Western countries, which are South Africa's largest financial partners, and especially for the Carter administration, which has built much of its innovative African policy around the idea of cooperating with key black African states such as Nigeria, Tanzania and Angola in trying to resolve the racial and political conflicts in Rhodesia and Namibia.

Angola and Nigera were instrumental in pressing the South West African Peoples Organization (SWAPO into unexpectedly accepting the U.N. plan in September. SWAPO has been waging a low-level but escalating insurfency in Nambia for a decade. The delivery of SWAPO by the Africans came as a surprise to the South Africans, who then withdrew their agreement to the plan.

Following Rhodesian Prime Minister Ian Smith's visit to the United States this month, a failure by the Carter administration to get the South Africans to stand by their earlier agreement would expose friendly African leaders to intense domestic criticism and could produce important policy shifts, according to knowledgeable African officials.

Suggested late last month to Vance by West German Foreign Minister Hans-Dietrich Genscher, the mission was originally intended in large part to impress Botha with the West's inability to oppose credibly the demand by Africans for economic sanctions if Pretoria goes ahead with the December elections.

Western vetoes in the Security Cuncil have traditionally protected South Africa against resolutions calling for cutting off all its trade. Western commitments to continue negotiations on Namibia with South Africa were cited as reason for such vetoes in 1977.

But the five nations appear to have come to the talks with less of a unified front than in the earlier negotiations. They also appear to be taking a softer approach toward Botha than originally intended. U.S. official talk of trying to "letter rather than flatten" Botha into reaching a face-saving compromise.

French Foreign Minister Oougs de Gutringaud declined to come because he did not think that the South Africans caould be talked into such a compromise, diplomattic sources report. Deputy Foreign Minister Olibier Stirn is representing France.

Botha, who brought South Africa's military commander-in-chief, Gen. Magnus Malan, into the talks, met with Vance and his colleagues for nearly two hours yesterday morning. The five foreign ministers then continued talks with their South African counterpart, Pik Botha.

In another sign of a softening of the Western stance, the contact group also met with Justice A. M. Steyn, the South African-appointed administrator general of Namibia. Earlier, U.S. sources had indicated that the five did not want to convey a special air of authority to Steyn by meeting him apart from the rest of the South African delegation.

Southwest Africa, about twice the size of California, was taken over by South Africa under a League of Nations mandate in 1920. It has about 100,000 whites and 800,000 black residents and possesses major deposits of diamonds, uranium and other valuable minerals.

The principal objections raised by the South Africans to the U.N. plan which is to be reviewed by the Security Council on Oct. 23, concern the delay of seven to eight months in elections that the plan calls for and the proposed stationing of 7,500 peace-keeping troops in the territory to oversee the elections after there is an effective cease-fire with SWAPO.