Britain has abandoned hopes of a quick peace conference among Rhodesia's contending black factions, it was learned yesterday.

The still-born proposal, agreed on only last week by President Carter and Foreign Secretary David Owen, called for a meeting next week of guerrilla chiefs and the Rhodesian nationalists who have reached agreement with Prime Minister Ian Smith. According to authoritative sources here, however, the plan has now been shelved.

Owen has spent much of the past two days talking with the guerrilla leaders, Joshua Nkomo and Robert Mugabe of the Patriotic Front. They emerged from a meeting yesterday saying they would not talk to anyone except U.S. and British representatives.

According to the best-informed officials here, however, the problem is not who takes part in talks but who controls police and military weapons during the transition period to black majority rule.

In London's view, the deal worked out with Smith leaves this control firmly in white hands. That is a major reason why neither the United States or Britain has endorsed the internal pact.

For their part, Nkomo and Mugabe have not retreated from their insistence that they must enjoy a monopoly of legitimate force. This is unthinkable to Smith and his new black allies, Bishop Abel Muzorewa and the Rev. Ndabaningi Sithole.

In Salisbury, Rhodesia, Sithole and Muzorewa and the third nationalist leader, Chief Jeremish Chirau, said they were rejecting the proposal for talks with Mugabe and Nkomo.

The British government thinks the stalemate means that the guerrilla war will go on until either one side or the other wins outright or an endless deadlock will induce a spirit of negotiation.

The United States and Britain say they are still attached to the original anglo-American settlement plan, and believe it must reemerge as the only realistic alternative to a civil war.

That plan tried to compromise the Smith-Patriotic Front positions by giving armed forces of both camps a role in the transition, adding a neutral United Nations force to the mix and putting all armed forces under a British commissioner, Field Marshall Lord Carver.

Smith, however, is satisfied with the deal he has struck with the three black leaders inside Rhodesia. For their part, Nkomo and Mugabe reject any use of Smith's police and military, any U.N. force and any command by Carver. Mugabe, morever, is even resisting the Anglo-American call for an amnesty on both sides. He wants to prosecute white Rhodesian officials he thinks have committed crimes against blacks.

Apart from leaving guns in white hands, the arrangement written in Salisbury allows Smith to continue as prime minister and his white Parliament to remain in power until elections are held. Then, in London's view, it provides for a continuing white veto over crucial issues like land reform.

All this, it is explained here, compels London and Washington to sustain their uncomfortable stance, neither endorsing nor condemning the internal pact. Owen and the Labor government feel strong pressure from opposition Conservatives and a largely Tory press to bless the internal deal, but insist they have no intention of doing so.