Britain said today that it would start talks with the biracial government of Prime Minister Abel Muzorewa Tuesday on measures to bring about legal independence for Zimbabwe-Rhodesia. Such a move is likely to lead to a deal that excludes guerrilla leaders and end chances for a peaceful settlement in the embattled country.

British Foreign Secretary Lord Carrington said the Patriotic Front guerrila alliance, the other party in the five-week-old settlement conference, would only be allowed to participate if it first agreed to a British-proposed constitution.

Front leaders Robert Mugabe and Joshua Nkomo turned down the constitution for the third time in a week in a stormy meeting with Carrington this morning.

Carrington, chairman of the conference, is challenging the guerrillas to either accept the constitution or quit the talks.

Negotiations limited to Muzorewa's Zimbabwe-Rhodesian government, which has already accepted the constitution would be almost certain to lead to a separate deal, leaving out the Patriotic Front.

The Front has warned repeatedly that a deal excluding the guerrillas would lead to an escalation of the seven-year-old war with the possibility of increased Communist support which could result in an East-West confrontation in southern Africa.

The problem for the British is whether any deal with Muzorewa alone would win sufficient international support to allow the lifting of economic sanctions and recognition of the government, which has been a pariah since the former white-minority regime illegally declared independence in 1865.

The first sign of difficulty arose tonight when the Commonwealth, under whose auspices Britain is holding the conference, sharply criticized Carrington's move.

Sridath Ramphal, secretary general of the 41-nation organization of former British colonies, said the Commonwealth had agreed at its summit meeting at Lusaka, Zambia, two months ago that an agreement had to include all parties. He added, "It would be a mistake to assume Commonwealth support for any procedure at variance with" the Lusaka accord.

Leaders of the African "front-line-states" supporting the guerrillas scheduled a summit meeting Tuesday in Dar es Salaam, Tanzania, amid reports that they were unhappy with Carrington's take-it-or-leave-it tactics.

British and U.S. diplomats have been lobbying for support of the constitution among the five front-line nations -- Zambia, Mozambique, Tanzania, Angola and Botswana. Western Muzorewa could seriously affect relations with black Africa.

The Patriotic Front limited its immediate response simply to saying, "The war will continue," Emphasizing the gravity of the situation, Mugabe and Nkomo met late into the night to work out a response.

Muzorewa's foreign minister, David Mukome said he welcomed Carrington's move to get on with the conference. Zimbabwe-Rhodesian military commander Gen. Peter Walls arrived here yesterday for talks about military aspects of a settlement, a signal that a possible separate deal was in the offing.

British sources described Carrington's 35-minute meeting this morning with the two Front leaders as "strained" and added that there were "sharp exchanges," Sources close to the Front went further, saying Carrington's attitude was "antagonistic."

A continuation of the hostile atomsphere will make it more difficult for the Front to rejoin the talks because it will be harder to find a face-saving formula.

Carrington attempted to ease the atmosphere at a press conference, his first during the conference. He said, "We very much hope that the Patriotic Front will agree to participate" in Tuesday's meeting.

He added that he was not abandoning hope of successfully concluding the conference and ending the war but repeated that the constitution proposed by Britain represented the only chance of a compromise between the Patriotic Front and Muzorewa delegations.

"We've come a long way in the last five weeks," he said, referring to compromises made by both sides in the constitution that remove controversial elements of authority for the white minority of 230,000.

The Front objects mainly to articles that entrench white rights to compensation for land and pensions, provisions that could cost the new government more than $1 billion.

Britain and the United States have both offered to participate in an unspecified agricultural aid program to help solve the land problem.

Asked at his news conference how the war could stop without guerrilla agreement, Carrington said: "We must home that there will be international support for an agreement that implements a fair constitution under free elections under British authority with Commonwealth observers." With such support, he added, "I hope the war will stop."

Repeatedly throughout the conference, Britain has claimed to be carrying out the wishes of the Commonwealth.

Thus if the three-way talks end it will be of critical importance which party is seen as causing the breakup. Mugabe and Nkomo have said on several occasions that they came here to talk and have no intention of walking out of the conference.