Diplomats here said today that the U.S. search for a settlement that would get South Africa to grant independence to Namibia and result in Cuban troops leaving Angola has reached a deadlock and there is no longer any chance of an early settlement.

These sources, connected to the five-power Western "contact group" that has been led by the United States, said the search for a solution would continue. But their comments clearly indicated that the front of optimism that the Reagan administration has kept up in its 15-month search for a diplomatic triumph has collapsed.

The mood of optimism had been quietly encouraged in Washington even as two cease-fire deadlines of Aug. 15 and Sept. 15 passed. But the hopes that were voiced for the setting of new target dates have now foundered over South African insistence, after most matters on Namibia itself have been agreed, that the estimated 20,000 Cuban troops in Angola must agree to leave before the Namibia deal can be signed.

As the South African position on this point has hardened, Angola has backed away from the U.S. effort, which it had been seriously considering. When Frank Wisner, deputy assistant secretary of state for Africa, visited Luanda, the Angolan capital, earlier this month to discuss possible plans for a Cuban withdrawal, President Jose Eduardo dos Santos, who had seen Wisner on three previous visits, did not meet with him.

Since the Reagan administration began the current initiative just over a year ago, the United States has shared South Africa's desire to secure a Cuban withdrawal as part of a package deal on Namibia, but Angola has resisted any linkage of the two issues.

The United States remained hopeful of negotiating an informal understanding with Angola that the Cubans would start leaving once Namibia was independent, but on Sept. 13 South African Prime Minister Pieter W. Botha made it clear his government required something more specific.

Angola contends it needs the Cubans to protect it from South African troops based in Namibia, who make deep incursions into southern Angola to strike at guerrillas of the South-West African People's Organization (SWAPO), who are fighting a war of insurgency in northern Namibia.

Angola's ambassador to France, Luis Jose de Almeida, said Friday in Paris that his government was prepared to agree to a "token withdrawal" of some Cuban forces provided South Africa pulled its troops out of Namibia and ceased what he called aggressive acts and threats against Angola, including any assistance to opposition groups in Angola, Reuter reported from Paris.

Angola also argues that as a sovereign country it is entitled to seek aid from allies to help defend its territory and that their withdrawal cannot be made a condition for an agreement on the constitutional future of another country.

Other countries in the region back Angola on this, accusing South Africa of following a policy of military adventurism in their territories to destablize them.

Botha said recently that South Africa's "sphere of influence" extended to all its neighboring black states and that it would not permit a hostile foreign presence in any of them.

Foreign Minister Roelof F. Botha and Defense Minister Magnus Malan have made similar statements.

This view in Pretoria was amplified last weekend by the political correspondent of the Johannesburg Sunday Express, who quoted an unnamed senior government source as citing U.S. action in stopping Soviet missiles being installed in Cuba as a precedent that justified South Africa's taking preemptive action in its own "sphere of influence."

"We are not a great world power," the source was quoted as saying, "but we are at least the power to be reckoned with in southern Africa."

He contended that the introduction of hostile forces into neighboring states was not only a threat in itself, but also provided a protective umbrella for insurgents operating against Namibia and South Africa and South Africa would not hesitate to remove them.

"It does not matter if the communist military forces are there legitimately as allies of those states," the source reportedly said.

Such strategic considerations may not be the only reason South Africa has hardened its stance on the Cubans to the point where it has removed the prospect of a settlement on Namibia anytime soon.

Some political observers here say Prime Minister Botha may be trying to delay a settlement because of growing anxieties about a right-wing backlash in his own electorate over planned constitutional reforms in South Africa itself.

A Namibian settlement would almost certainly add to this backlash, particularly if SWAPO were to win the pre-independence elections, as most people in touch with the territory consider likely.

For years South Africa has portrayed the SWAPO insurgents as surrogates of Moscow. Now Botha finds himself accused by his opponents on the far right of preparing to sell out Namibian whites to these "Marxist terrorists." He may have decided it would be imprudent to run the political risks of internal reform and a Namibian settlement at the same time.

Yet, with an explicit undertaking on the Cubans, Botha could point to their departure as evidence that he is rolling back the communist threat his government says is facing South Africa.