The Carter administration has dispatched a senior diplomat from the U.S. mission to the United Nations to Angola for unannounced talks with Angolan President Agostinho Neto on ways to calm conditions on the Angola-Zaire border.

The decision to send this mission to Angola follows two months of sometimes intense debate within the Carter administration on possible American responses to Soviet-Cuban presence in several African nations, including Angola.

The plan to send Ambassador Donald McHenry to Angola was finally decided upon last Wednesday by Secretary of State Cyrus R. Vance, administration sources said. Angola and the United States agreed to make no public announcement of the visit, official sources said.

McHenry will also be discussing obstacles to a negotiated transfer to majority rule in Namibia, Angola's neighbor to the south, in his talks with President Neto and other Angolan officials, sources said.

Although the United States has been in regular contact with Angola over Namibia, the general relationship between the two countries has been tense and Hostile since 1975, when Neto invited Cuban troops into Angola to help him defeat western-supported factions in a bitter civil war.

The Neto regime is avowedly Marxist, and is supported by 20,000 Cuban troops.

In recent weeks the Carter administration assailed Cuban and Angola for abetting the invasion of Zaire's Shaba Province by Kantangese rebels who have been living inside Angola for years.

Just seven weeks ago, Central Intelligence Agency Director Stansfield Turner, acting by his account on instructions from a committee of the National Security Council, approached a key senator with a plan to supply covert American aid to rebel guerrillas who still fighting against the Neto regime in Angola.

President Carter's national security affairs adviser, Zbigniew Brzezinksi, was reportedly the originator of the idea to consider supplying covert aid to Neto's internal opponents in an effort to tie down the Cubans in Angola, making it difficult for them to move to other potential African battlegrounds, particularly Rhodesia.

The State Department, particularly the senior officials responsible for African policy, bitterly opposed this idea.

One of those senior officials said last night that the new decision to send an envoy to Angola to look for ways to deal directly and cooperatively with the Neto regime was a consequence of Carter's decision to rule out any convert U.S. involvement in Angola. (Such involvement is ruled out by law, so any attempt to aid Neto's opponents would have required congressional action.)

Carter announced that decision in a press conference in Chicago on May 25.

Since then administration planners have been looking for new options, and in recent days the president accepted Vance's ideas for approaches to the problems in Africa. The outline of these approaches was contained in a major speech Vance delivered Tuesday in Atlantic City, N.J.

In that speech Vance said Zaire and Angola "must reach agreement to respect their common border and not to interfere in each other's internal affairs."

This was a referee both to the Angolan sanctuary for Katangese rebels and Zairian support for rebel forces of the National Union for the Total Independence of Angola (UNITA) who use Zaire as their major rear base area in their fight against the Neto regime.

Several administration officials yesterday expressed the hope that Angola and Zaire might eventually come to an understanding that neither would permit its territory to be used to launch attacks against the other.

U.S. officials said yesterday that in previous contacts with Angola, the Neto government has invariably raised the issue of UNITA guerrillas using Zaire as a base. According to these officials, Angola appears to hold the United States responsible for Zairian policy, arguing that President Mobutu Sese Seko is dependent on the United States for his survival.

Despite the mutual recriminations that have typified past American-Angolan relations, the two countries have also quietly maintained contact, U.S. officials said yesterday. For example, during the Katangese rebels' invasion of Zaire's Shaba Province last month, when the United States was publicly critical of Angola, the two countries exhanged messages on possible ways to handle hostages the rebels might bring from Shaba back to Angola, the sources said.

The Shaba invasion also persuaded some U.S. officials that the previous American posture of rhetorical attacks on the Soviet-Cuban presence in Africa was not adequate to deal with practical problems on the ground such as the Katangese rebels' ambition to destroy the Shaba economy.

"We had the choice of either stiring it up or quieting it down, and if you wanted to quiet it down you had to talk to the Angolans," one official said.

Angola's President Neto may have come to a similar conclusion. Earlier this month, according to reliable sources, he sent three messages through different channels to the United States suggesting the possibility of Angolan-American discussions.

A ranking U.S. official said yesterday that this country had encouraged Neto to send those messages, having decided that it would make sense to try to negotiate with Angola.

Another aspect of the administration's new approach involves its much harder line on future aid to Zaire, sources said yesterday.

Secretary Vance spoke of the need for Zairian political and economic reforms as a condition to further economic aid in strong terms in his Atlantic City speech Tuesday.

Sources said yesterday that the United States has told Mobutu he won't even receive food aid under Public Law 480 without broadening his political base, taking steps to combat corruption and imposing economic austerity. According to these sources, the United States has taken a much harder line with Mobutu than other interested West European states.

Heavy pressure on Mobutu could have the side effect of helping persuade Neto that the United States is serious about its desire to stabilize the situation in the part of Africa on an equitable basis, one official source said.

State Department officials yesterday described the new African approach as a victory for their boss, Vance, in the administration's infighting.

Several members of Congress who attended a foreign policy discussion Tuesday night at the White House said yesterday they also felt Vance's star was rising in the administration, while Brzezinski's appeared to be in decline.

Carter, Vance, Brzezinski and Defense Secretary Harold Brown all spoke to the congressmen and answered their questions on a wide variety of issues.

Rep. Don Bonker (D-Wash.), a member of the House Committe on International Relations, said yesterday that at the White House meeting, "I got the impression that the reins are being pulled back on Dr. Brzezinski and are being let out a little for Secretary Vance."

Half a dozen others who attended the meeting said Vance played the leading role apart from Carter. Some thought Carter was deliberately building up his secretary of state, others said Vance's role appeared to reflect normal protocol.

According to participants in the meeting. Carter emphasized his desire to continue negotiations with the Soviet Union on strategic arms control despite any other events in Africa or elsewhere.