The U.S. government, apparently reluctant to annoy South Africa before a settlement is reached on Namibian independence, reportedly refused this week to join a French-initiated move to have the diplomatic corps here formally protest handling of the black squatter situation in Cape Town.

Opposition by the Americans, who were later joined by the initially wavering West Germans and British, deprived the move of support of three influential backers and helped kill it, according to confidential papers shown to this reporter.

U.S. Charge d'Affaires Howard Walker said today he had no comment. "Any such conversations, if they went on, would have been private and by that I am not saying they went on," he said.

[In Washington, a State Department spokeswoman also declined comment.]

More than 1,000 black squatters who have been claiming the right to live together as families in Cape Town were arrested this week and forcibly returned to their homeland of Transkei, where jobs are scarce. The move has angered whites as well as blacks.

The action followed a month of resistance by the squatters to government orders to leave the area. During that time police made more than 1,100 arrests and demolished plastic shanties donated by churches and other groups, leaving families without shelter in the cold and rain of Cape Town's winter.

Foreign Minister Roelof "Pik" Botha defended his government's handling of the squatters in a luncheon yesterday with foreign correspondents.

The squatters "are Transkei citizens. They are not South African citizens. They are like Mexicans in the U.S. . . . and when they enter the U.S. illegally, I will show you what happened to them. I know because I had a Mexican maid who worked for us at one stage," said Botha, who was once ambassador in Washington.

Botha implied a separate standard was applied to South Africa because governments around the world acted against squatters, "but when we in South Africa are faced with a similar dilemma, look what happens to us."

But the significance of the squatter controversy goes beyond the police action to one of the underpinnings of apartheid -- the migratory labor system and "pass laws" that regulate the number of blacks coming to the white-controlled cities and often force breadwinners to live apart from their families.

Beyond highlighting how the Namibia issue is now the critical touchstone for the Reagan administration's policy toward South Africa, the reported diplomatic discussions reveal a divergence of approach between the Socialist French government and Washington.

If the cleavage grows, it may pose serious problems within the "contact group" of five Western nations trying to draw up settlement proposals on Namibia, now administered by South Africa as South-West Africa, that are acceptable to both Pretoria and its guerrilla foe, the South-West Africa People's Organization (SWAPO).

Already some Western diplomats here have remarked on serious strains in the contact group, which also includes West Germany, Britain and Canada, over the Reagan administration's efforts to try a soft-sell approach with Pretoria and go back to basics in the already lengthy Namibian negotiations in the hope of getting Pretoria's cooperation on a settlement.

Although many blacks here were disillusioned with the Carter administration in its later months, and many admitted its anti-South African rhetoric accomplished little and sometimes even backfired, they are angered by the present administration's lack of public criticism of Pretoria, and by its decisions to increase the number of defense attaches and to grant visas to the South African nonblack Springboks rugby players.

While some blacks commenting on the issue are prepared to give the Reagan policy a change, they would also like to see that engagement extended to blacks also. Zulu chief Gatsha Buthelezi told a visiting congressional delegation here two weeks ago that he would like to see more than "symbolic recognition" for the "external liberation movements" and some assistance in "practical ways" to leaders like himself working internally for change.

According to a reliable source who provided documentary evidence, French Ambassador Bernard Dorin, an acting dean of the diplomatic corps in Cape Town (South Africa's legislative capital), called together the heads of mission Aug. 13.

The group reportedly drafted a protest saying, "We the heads of mission wish to express our concern about recent events in the Nyanga Crossroads area which are increasingly engaging the attention of our governments and people. We appeal to the South African government to give humanitarian considerations high priority in the handling of the situation."

Britain, Canada, West Germany and the United States reportedly asked to refer the matter to their capitals over the weekend. When they met again Monday, Walker did not attend, sending word that the United States declined to participate, according to the papers.

By this account, Britain and West Germany were having doubts on Monday. By the end of the week Bonn sent a negative response to its ambassador here, and Britain said it would only go along if all 10 European Community nations did -- which was seen as a refusal to participate.

Among those who declined from the start were Argentina, Paraguay, Portugal, Greece, Spain and Uruguay. The countries that reportedly might have gone along, had the United States and its two major allies done so, were Japan, Canada, Austria and Italy.

By the end of the week only Australia, Belgium, Denmark, Finland, Malawi and Sweden were prepared to join France.

Word of the move reached the South African government, the papers indicate. On Thursday, the Portuguese ambassador, back at his post as dean of the corps, received a message from South African Secretary of Foreign Affairs Brand Fourie saying that although his government recognized the right of government to make bilateral demarches, any joint representation by the heads of mission without prior authorization from their governments could "lead to complications."