The United States and its four main Western allies urged the United Nations yesterday to avoid an immediate confrontation with South Africa despite the failure of the Western nations to win agreement in Pretoria this week for a U.N. plan for independence for Namibia.

The request for a new delay in punitive U.N. action - which could come as early as Monday and could force the West to choose sides in a costly economic war with South Africa - was spelled out in a joint communique issued here by Secretary of State Cyrus Vance and in Pretoria by Prime Minister P. W. Botha.

Vance described the communique as a "step forward" but offered no predictions on U.N. or black African reaction to it. The document amounts to a temporary compromise to keep talks going in the Namibia dispute, and Vance acknowledged that the five Western countries have failed to win any assurances from the South Africans on a number of key points.

In Pretoria, however, the South African government sought to put a positive gloss on the communique, suggesting that it opens the way for implementing a U.N. plan for elections and transition to independence for Namibia, Washington Post correspondent Caryle Murphy reported. Namibia is also known as Southwest Africa.

The South Africans canceled on Sept. 20 their earlier agreement to a plan for U.N.-supervised elections and independence for Namibia. Instead, they said they would hold "internal" elections in Namibia in December in an obvious effort to deny any political role to the leftist Southwest Africa People's Organization (SWAPO), which is conducting a guerrilla war inside Namibia.

Vance said at a news conference that the South Africans still intend to go ahead with unilateral elections in the territory in December and that they did not clearly commit themselves to supporting internationally recognized voting later.

Speaking in Pretoria. Botha said for the first time that if SWAPO won the U.N.-supervised elections, his government "would accept the verdict of the people of Southwest Africa."

But Botha would not define what authority the winners of the December "internal" elections would have, nor would he give a firm view on Pretoria's position in case these elected leaders refused to cooperate with the U.N. plan.

Vance, who left Pretoria Wednesday after three days of difficult negotiations with Botha, reached the limited measure of compromise only after an exercise in extreme brinkmanship by the South Africans, according to U.S. officials.

When Vance and his counterparts from Britain, Canada, West Germany and France prepared to leave the government building in Pretoria Wednesday, they were convinced that the South Africans had chosen to announce publicly that the talks had ended in failure.

In saying goodby, however, Foreign Minister Pik Botha suddenly began a lengthy discussion with Vance on the limited compromise proposal. This in turn led to the South African decision to agree in the final communique to a new round of talks with a special representative of U.N. Secretary General Kurt Waldheim over Namibia.

Waldheim is due to report to the Security Council on results of the 16-month-old negotiations between the five-nation "contact group" and Pretoria on Monday. A report of total deadlock on Namibia was widely expected to trigger demands from African states for trade embargoes against South Africa that would have posed threats both to Britain's economy and to the Carter administration's diplomatic campaign to win over black African nations.

The failure of the talks to produce a clear-cut result left the South Africans and the five delegations free to interpret the points covered in the communique in widely differing ways. News agencies reported that British, German and South Africans publicly hailed the results as a major success, while Vance was much more restrained in his description of the communique, which he noted contained "sharp differences in views" on the election issue.

London, Canadian Foreign Minister Donald Jamieson made it plain that the real objective of the compromise formula was to avert U.N. sanctions against trade with South Africa, Washington Post correspondent Bernard D. Nossiter reported.

Jamieson told reporters the five Western foreign ministers regarded their "agreement" with Pretoria as "sufficient progress" to postpone both a Security Council debate as well as Security Council action against South Africa.

But he acknowledged that no agreement had been reached on the central objective sought by the five, either postponement or dilution of the elections for a Namibian government under Pretoria's auspices. The formula spells out South Arica's hope that the government to be elected in December will somehow gain world approval.

The compromise appears to strike the middle ground between Pretoria's demand for an endorsement of the December elections and the West's demands for guarantees from South Africa that those elections, if held, would be treated as nothing more than a test of public opinion.

The agreement to begin a new round of talks was coupled with an appeal by the five nations to Waldheim to send his representative, Martti Ahtisaari, to the provincial Namibian capital of Windhoek "as soon as possible" to open new talks with South Africa's administrator general, Justice M.T. Steyn, on fixing a date for bringing U.N. troops into the territory and for elections that the U.N. will agree to supervise.

Vance said the five nations would be in immediate contact with African states that support the SWAPO guerrillas "to urge them to let the process we are recommending go forward." If this first step is successful, a confrontation next week in the Security Council could be headed off and Waldheim would dispatch Ahtisaari for new talks.

U.S. officials said yesterday those talks would then provide a final measure of whether South Africa is now determined to push ahead with an internal settlement that would risk continuing the guerrilla war and international sanctions or would move back to supporting an internationally recognized solution even if it means giving SWAPO a chance to win power eventually.

The communique commits Botha's government only to "using its best efforts to persuade" the locally elected leaders "seriously to consider ways and means of achieving international recognition through the good offices" of Ahtisaari and Steyn.

The differences over the elections take up three of the joint communique's five brief paragraphs. The two other paragraphs cover what Vance called "agreement in principle" on the number of U.N. troops and the countries they will come from, and limiting the role of a civilian U.N. police force to monitoring the existing internal security network South Africa had specifically cited its distrust of U.N. and Western interpretations of these points in pulling out of the earlier agreement in September.

South Africa also insisted on issuing a separate statement covering its concern that SWAPO, which was not mentioned by name, could hold off the second set of elections indefinitely by refusing to abide by a firm cease fire until it was sure of winning the elections. Vance said the West's statement, which committed the five countries to establishing small observer forces of their own in the territory and to taking action to back up a cease-fire, "completely answers" that convern.

Vance stopped off in Geneva yesterday before flying to Moscow today for strategic arms limitation talks.