American-led efforts to negotiate a peaceful settlement of the guerrilla war in Namibia appeared to be close to collapse yesterday after the Southwest Africa Peoples Organization (SWAPO) rejected a compromise plan for new talks between the United Nations and the South African government.

The rejection was contained in a letter handed to U.N. Secretary General Kurt Waldheim in New York by SWAPO representative Theo Ben Gurtrab, according to informed sources. The letter is due to be made public later this week.

Waldheim also met yesterday with ambassadors representing the "frontline" African states that provide logistical support to the guerrillas, and again heard strongly negative comments about the compromise proposal, these sources reported. The five frontline states, and particularly Angola, have been crucial to the U.S. efforts for a peace agreement.

The decision by SWAPO and the African states not to make their separate rejections of the compromise public and formal at this point left U.N. officials clinging to a slender hope that Waldheim could still work out a formula to keep Namibia talks going and to avoid calls in the Security Council for international economic warfare against South Africa.

U.S. officials declined to comment immediately on yesterday's move by SWAPO, but expressed hope that a U.N. economic confrontation with South Africa can still be avoided. There were suggestions in the comments by U.S. officials that the Carter administration is pressing Waldheim to send a special representative to talks with the South Africans despite the initial negative responses from black Africa.

The United States and other Western nations have in the past vetoed U.N. economic sanctions against South Africa, which has ruled the largely desert territory of Southwest Africa (now known as Namibia) as a mandate area since 1920.

Avoiding a sanctions debate was one of the chief objectives of a five-nation negotiating delegation, headed by Secretary of State Cyrus R. Vance, that went to Pretoria last week to get Prime Minister Pieter Botha to agree to hold U.N. supervised elections in Namibia and to withdraw South African troops.

Botha agreed only to reopen talks about U.N.-supervised elections with Waldheim's special representative, Martti Ahtisaari. In a joint communique, Botha declared that South Africa would go ahead with its own elections in the territory Dec. 4.

SWAPO, which is receiving increasing support from the Soviet Union in its low-level, decade-old insurgency against South Africa rule, attacked the joint communique in its letter to Waldheim as a plan to install a puppet regime in Namibia and said it opposed new talks with Pretoria, according to informed sources.

Before meeting with Waldheim, Ben Gurirab told reporters that SWAPO would press for an early meeting in the Security Council to impose "comprehensive, mandatory economic sanctions" against South Africa. U.N. sources said African nations were preparing requests for specific trade embargoes, including petroleum.

The five-nation Western group obtained agreement from South Africa and from SWAPO over the summer for a U.N. plan for independence for Namibia. But South Africa withdrew its agreement Sept. 20, charging that the United Nations was changing details of the plans that would weight the elections in favor of SWAPO.

During last week's negotiations, Botha said he was now satisfied with new Western undertakings on the specific details in dispute, but insisted the December elections under South African control had to go ahead.