The tangled Rhodesian conflict took another unexpected turn yesterday when Zambia's President Kenneth Kaunda withdrew his support from an Anglo-American plan aiming to bring a peaceful transfer of power in Salisbury next year.

Kaunda's announcement appeared a clear setback for prospects for peaceful settlement of the Rhodesian deadlock. Zambia was one of the five "frontline" African states that had earlier given qualified approval for the Western iniative to establish a black majority rule in Rhodesia.

In an equally unexpected development, two key leaders of the Rhodesian Patriotic Front, the militant black nationalist organization whose guerrilla forces are fighting the white minority government, yesterday apparently parted ways over how to deal with Western efforts for breaking the deadlock.

While Joshua Nkomo accepted a British invitation to attend talks on Rhodesia in London next week, Robert Mugabe turned it down. Nkomo and Mugabe are co-chairmen of the front and rivals for control of the movement.

An additional dose of confusion was added by diplomatic reports that the two Western powers also are at odds over how to proceed with efforts to end the war in Rhodesia.

Following the latest moves by Prime Minister Ian Smith, the British are said to be more inclined to give his white-mminority government a chance to reach settlement with moderate black leaders living in Rhodesia. U.S. sources here believe that a political settlement that would exclude the Patriotic Front would not bring an end to the guerrilla war inside Rhodesia.

Kaunda's action yesterday was regarded by analysts as most significant. By withdrawing from a Western diplomatic initiative, the Zambian leader broke ranks with other "frontline" states - Tanzania, Angola, Botswana and Mozambique - and presented London and Washington with further complications in an already complex situation.

American officials here sought to emphasize that Kaunda did not reject the Western plan and that while withdrawing had asserted he would not try to block it.

The Zambian leader, seeking at a press conference at Lusaka, denounced Rhodesia's military incursions into Mozambique and Zambia and expressed the belief that Rhodesian forces were actively supported by South African troops.

"We now say we withdraw from the debate," he said. "We will no longer sit around a conference table to discuss the proposals. We will not again make any proposals. We will not discuss the Anglo-American initiative. We will not participate, let them discuss it themselves."

He made it clear that Zambia would continue to support Patriotic Front guerrillas who are waging armed struggle against Rhodesia from bases in Zambia and Mozambique.

At the same time, a top aide to Rhodesian nationalist Bishop Abel Muzorewa has been given seven days to leave Zambia, where he had been based for 13 years. Muzorewa's United African National Council is a rival group to the patriotic Front, which Zambia recognizes.

Reports from Salisbury suggested yesterday that the apparent disunity among the frontline" states as well as the seeming split in the Patriotic Front would help Smith in his efforts to reach an internal settlement with moderate black leaders.