The South African government announced today that it would join with the United States and Britain next week to try to break the deadlock over Rhodesia negotiations with a new initiative.

Foreign Minister Hilgard Muller said "there are great expectations" about the talks. It appears to mark the first time that South Africa has taken the initiative in the negotiations.

The plan reportedly backed by South Africa - the only country with the ability to pressure Rhodesia into a settlement - would involve the white-minority government's efforts to deal directly with moderate nationalists without foreign participation.

The South Africans are apparently taking great care to do nothing without Anglo-American backing so that Pretoria cannot be accused of being alone in leaning on the Rhodesians. It is thought that any proposal that emerges from the new talks - expected at the U.S. State Department in Washington on Monday - will be carefully labeled as a joint Anglo-American South African initiative.

The key to acceptance of this plan, known in Rhodesia as the "internal "option," would be a national referendum to determine which of the four rival nationalist leaders holds the majority of black support.

It has been clear since the Geneva peace talks between Rhodesia's white government officials and four black nationalist leaders adjourned in mid-December that the three governments were desperately searching behind the scenes for a way to resolve the Rhodesian crisis peacefully.

There has been growing speculation about another try since South African Prime Minister John Vorster and Rhodesian Prime Minister Ian Smith held a surprise summit in Cape Town last week. The next day Vorster called in U.S. Ambassador William Bowdler and British Ambassador Sir David Scott for discussions.

On behalf of the three governments, British U.N. Ambassador Ivor Richard spend four weeks shuttling between white and black Africa in January, attempting to sell an alternative peace plan to the original proposals formualted by former U.S. Secretary of state Henry A. Kissinger. Richard has recently been quietly seeing officials in Washington. But the plan - which called for a black-dominated transition government - was rejected on Jan. 24 by Smith.

Smith has made it clear that he would like to include Bishop Abel Muzorewa, the most moderate of the four nationalists, in the "internal options" negotiations. At this point it appears that Muzorewa, an American-educated Methodist, also holds the support of the majority of blacks inside the country - although his strength among the guerillas is unknown.

An election proving Muzorewa's strength would be the only means of getting the bishop, who has so far balked at bilateral talks with the Rhodesian leader, to participate. But there are two drawbacks to outside recognition of this effort:

The acceptability of elections run by the illegal Rhodesian government. The three governments may be considering outside supervision of a referendum.

The role of the Patriotic Front - the alliance of militant nationalists Robert Mugabe and Joshua Nkomo - who have the support of the guerrillas and the five "front-line" African presidents involved in the settlement effort.

It appears that the Smith government would, at a minimum be reluctant to let the two leaders back in the country for an election since they are self-admitted guerrilla spokesmen.

Mugabe and Nkomo also have recently denounced the election idea and said they would not participate. Exclusion of the Patriotic Front or putting them in a back seat based on anticipated election results, could lead to rejection of the new settlement plan by the "front-line" presidents and an escalated guerrilla campaign.

There have been no indications from any of the governments on how this dispute could be resolved, although it is the crucial point in obtaining outside African support for a settlement and ending the war.

Officials in Salisbury have maintained that they are still willing to negotiate a settlement - although only on the basis of the Kissinger plan. It will take renewed pressure from South Africa - which is Rhodesia's economic lifeline to get the Smith government to move beyond those proposals under a new initiative.

Vorster has repeatedly taken a firm stand in public over the past three weeks against pressuring Rhodesia, which he called 'dishonorable.' But Muller's statement today indicates that the Vorster public stand may not be the final word.