The Rhodesia peace conference, which had been on the brink of success, was imperiled today when Patriotic Front guerrilla leaders withheld their signature from an agreement to end the civil war in Rhodesia.

Their refusal stunned British officials who thought they had removed all remaining sticking points in the 14-week-old peace conference here and had called today's final formal session to push Front leaders Robert Mugabe and Joshua Nkomo into signing a cease-fire.

Although the former biracial Salisbury government delegation joined the British in initialing a final agreement today and then left London for Rhodesia, British officials began a last round of informal meetings with Patriotic Front officials to try to persuade them to agree.

At Nkomo's suggestion, most of these meetings will be military talks between British and Patriotic Front generals about the number and location of the places where the guerrillas would have to gather to observe a cease-fire in Rhodesia. Nkomo said that was the last remaining obstacle before final agreement by the Patriotic Front.

If the guerrilla leaders do not sign a cease-fire, British officials made clear again today, the British governor already in Salisbury will go forward with new elections leading to black majority rule and legal independence.But the political parties headed by Mugabe and Nkomo would remain banned "because the Patriotic Front remains belligerent," said British spokesman Nicholas Fenn.

Patriotic Front spokesman Eddison Zvobgo answered by challenging the British to go ahead. "It would then be a British war against us," he said. "Let there be no mistake about that."

British Foreign Minister Lord Carrington was described by aides as "puzzled but not despairing" about the impasse. British officials still hope that the Patriotic Front would sign the peace agreement after Carrington and Prime Minister Margaret Thatcher return from an official visit to the United States next week.

But they noted after having been pushed by the British into agreements at earlier stages of the peace conference here, are faced with the momentous final decision of giving up the war, convincing their guerrillas to stop fighting and testing their political strength in an election.

Mugabe and Nkomo do not want to be seen by their supporters as having lost the guerrilla war or to be knuckling under to British pressure here. At previous critical junctures in the peace talks, they first angrily rebelled against similar British ultimatums only to agree to what they could later describe as a British concession to their demands.

The British thought they had made just such a concession today when Carrington promised that if more guerrillas emerged from hiding than the British had expected in alloting them 15 assembly places scattered throughout rural Rhodesia, the British governor in Salisbury "will assess the need for additional areas."

"If more people assemble than we thought would," a British source said later, "we would put it right."

Britian estimates from its own intelligence that thre are about 17,000 armed guerrillas inside Rhodesia. The Patriotic Front claims to have more than 35,000.

Patriotic Front spokesmen also claimed that by moving to the assembly places, the guerrillas would be giving up territory they had won in battle. British officials said the assembly places were chosen as safe locations for the guerrillas to gather and should not be moved to satisfy the Patriotic Front's political aims.

One ranking British military expert said he could understand the reluctance of Patriotic Front generals to give up ground they had fought for during seven years of guerrilla war. But other British officials pointed out that the United States and other NATO countries familiar with Britain's final proposals believe they are a fair compromise and should be accepted by the Patriotic Front.

U.S. officials have strongly supported the British at the peace talks here and have been pleased with what already has been accomplished. Today the United States announced it is following the British lead and lifting sanctions imposed on Rhodesia when the colony broke away from British rule.

But the failure to reach final agreement would force President Carter to make difficult decisions about continuing to back Britain's determination to go ahead anyway with new elections and the granting of legal independence to Rhodesia.

Mugabe complained today that "the British government has not treated us fairly" and tried to railroad the Patriotic Front into agreement by deciding "to send a governor to Salisbury prematurely."

Nkomo, however, was noticeably less negative and suggested that British and Patriotic Front "military men sit together" to negotiate the assembly points for the guerrillas without participation by "politicians who tend to talk too much."

Speculation persists here that Mugabe's wing of the Patriotic Front is divided over the possibility of ever agreeing to a cease-fire and that some of his guerrillas might never obey an order to stop fighting. Mugabe was the more reluctant of the two guerilla leaders to agree to a cease-fire in principle last week.

Carrington told Nkomo and Mugabe at today's final formal conference session that the British "do not believe that others would understand a decision by any party to continue the war against a lawful authority established to enable elections to be held in which all parties can participate. It will be a matter of grave disappointment to everybody if it is not possible to reach overall agreement at the conference after all we have achieved."

Nkomo, like Carrington, today took pains to point out that agreements already had been reached here on an independence constitution guaranteering black majority rule, on the holding of new elections, and on "Concept of a cease-fire." He said the "placement" of the guerrilla forces after a cease-fire was the last remaining hurdle.

Patriotic Front spokesmen said the guerrilla leaders would stay here to "continue negotiating" while British spokesman Fenn said "the door remains wide open to the Patriotic Front." mporary home since Dec. 2. In a three-hour talk they discussed the Pana