The Carter administration's diplomatic assault on southern Africa poses a serious test for another major foreign-policy goal it holds, that of closer cooperation with American allies in Europe.

An initial burst of cooperation has produced the first united front of Western nations, and some progress in preliminary negotiations, on the ticking racial time bombs of southern Africa. Holding the front together is vital to hopes for more progress, say American officials involved in the effort.

But conflicting interests and styles in southern Africa could tear the negotiating effort apart and, moreover, leave deep scars on the American transatlantic partnership with Britain and France, other officials fear.

Some of those strains are surfacing already. The repeated critical remarks by Andrew Young, U.S. ambassador to the United Nations, about Britain's "colonial mentality" are creating concern in London about a latent anti-British sentiment developing within the administration.

Many of the strains to date appear to be rooted in style. The younger, more moralistic American team working on southern Africa is impatient with Britain's seemingly "fuddy duddy," overly legalistic approach.

Britain and France, for their part, are having trouble digesting a new approach from Washington that the southern Africa initiative exemplifies. After having struck the highest possible profile on the issue, the administration is now following up with quiet negotiations desinged to get whites in Rhodesia and Namibia (Southwest Africa) to share power with blacks.

"Henry Kissinger really did do everything step by step, not just the Middle East," said an American official. "He believed you could only take one bite out of them at a time.

"The administration starts by announcing everything it wants, and loudly. You identify your goals, get people to focus on them, and then you pull back to start negotiating for what's obtainable right away, keeping the ultimate goal always in sight."

Kissinger - who used to say the most Americans thought Namibia was a make of automobile - was converted only late in his government service to paying any attention to Africa. The Carter administration is moving at a much more rapid and at times uneven pace.

Underlying European unease at the newly toughened American position toward South Africa is the specter of the United Nations attempting to impose economic sanctions against South Africa if the Nambia negotiations break down.

Britain, with three times as much investment in South Africa as the American stake of $1.5 billion, and France, which has followed up years of lucrative arms sales to Pretoria by now being given major contracts for commercial airliners, nuclear plants and other industrial goods, would undoubtedly veto the sanctions effort.

But they year that the United States might not join them. Young fanned the fears by telling an interviewer in London Thursday that, while "I would love to avoid [sanctions]," they might be imposed if "tied to specific aims," such as limiting new investment for mixed periods.

Young's wide-ranging trip to Africa this month and Vice President Mondale's jarring verbal confrontation with South Africa's prime minister, John Vorster, in Vienna last week helped establish another early, important difference in the American and British approaches to Rhodesia.

While he did not spell it out explicitly in public during the trip, Mondale is known now to have shaken Vorster with a toughly worded American refusal to support in any way a final agreement that entrenches the white minority's position, on a racial basis, once independent is achieved.

Kissinger was prepared to use a $100 million development fund to buy out farms of white settlers after independence and to support a scheme to give the whites an effective veto over the constitution. Although Britain still leans toward such protection as a way to make it easier for Prime Minister Ian Smith to come to terms, the United States says that the whites should not get special guarantees.

The British have been confused by conflicting signals from Washington on the importance the United States attaches to foreign secretary David Owen's new effort on Rhodesia.

After initially endorsing it as a joint approach and promising to name a senior official to represent the United States, Washington pulled back into a position of "supporting" a British "initiative," and named Steven Low, a competent but not well-known career diplomat as the American representative to the "consultation team." Officials in London noted that Low is currently serving as ambassador to Zambia, a country bitterly opposed to Smith's rule.

Details of the team's overall strategy have been tightly held, but it appears that the Anglo-American representatives will sketch the outlines of a constitution and draft the legislation that would bring it into being after talking to Smith, Rhodesian African nationalists and their principal backers in Africa.

The effort is to bypass the Geneva conference on Rhodesia, which broke down earlier this year, and to maneuver around extremists on both sides by not giving them the right to veto parts of the proposed constitution.

The team will reportedly try to get the detailed legislation it hopes to draw up accepted by the nationalists and then passed at the winter session of Rhodesia's parliament. Elections would be next srping, with a military force drawn from commonwealth nations in Rhodesia during the elections.

Nothing Mondale heard at Vienna suggested that Vorster opposed this approach. This inspired optimism on Rhodesia in the American delegation.

Voster clearly took a step back on Nambia, however. He had agreed in principle in talks with representatives of five Western nations in Cape Town early in May to free elections for a constituent assembly in Namibia with U.N. participation.

This would have effectively shelved the constitution that a South African-sponsored group of Africans and whites has been drawing up at the Turnhall conference in Windhoek and put that group on an equal footing with other political groups in the disputed territory.

At Vienna, however, Voster suddenly insisted that the Turnhall group should form the interim authority that would administer the territory during the elections. The United States suggested it could not accepted this, holding out for a more "neutral" administrative authority.

Vorster has agreed to meet again with the representative of the "contact group" composed of diplomats from the United States, France, Britain, West Germany and Canada.

The South Africans sought in the first round of talks in May to split the group, according to American sources, and are expected to play again on the large financial stake the British and French have in taking a softer attitude.

But American diplomats are convinced that countervailing pressure from black African states will keep the two European countries on board the negotiating effort.