The Rhodesia peace conference is like walking down a cul-de-sac with the British putting up barriers behind the other two parties to prevent them from turning back" as they successfully pass each point in negotiations.

This British method, described by an African diplomat close to the three-week-old talks seeking to settle the intractable Rhodesian issue, may get its biggest test next week when the warring sides move on to the issue of how to end the fighting and hold elections for an independent government.

The key question is what transitional arrangements Britain proposes as it attempts to maintain the image of steering a middle course between the Zimbabwe-Rhodesian government and the rival Patriotic Front guerrilla organization.

Britain, sponsor of the conference as the theoretically legal authority in its breakaway colony, has not tipped its hand so far.

The Patriotic Front, however, gave a small but significant indication this week that it is willing to be flexible on the transition period to implement a constitution -- an area where the parties are slowly moving toward grudging agreement.

Guerrilla commander Josiah Tongogara, the enemy most feared by white Rhodesians, said that if agreement is reached at the conference, he could work with the white military leadership.

"There is no reason why I should not work with Peter Walls. I am not afraid of working with (him) in any capacity," he told BBC radio. Walls is head of the Zimbabwe-Rhodesian military and the man responsible for devestating attacks on Tongogara's forces in neighboring Mozambique.

In Salisbury today, military spokesmen confirmed that Zimbabwe- Rhodesian forces are still operating in neighboring Mozambique, three days after beginning their first offensive against Patriotic Front guerrillas since the London settlement talks began earlier this month.

To many observers, the Front appeared to be sending out signals that it is willing to be moderate in the second phase of the negotiations and not demand a turnover of power or control of the military during the transition, as it has in the past.

It has indicated it is willing to share power equally with Zimbabwe-Rhodesian Prime Minister Abel Muzorewa and allow an international force to control the military situation during the transition.

Any show of reasonableness by the Front, led by Robert Mugabe and Joshua Nkomo, puts the Muzorewa delegation in a difficult position since it has little -- if any -- room to negotiate on the transition. Any move to set up a traditional government means that the bishop is negotiating away the very existence of his government.

"The British cannot ask us to cut our throats," a white Zimbabwe- Rhodesian official said.

A Front official, on the other hand, said the Muzorewa government cannot be in charge of the transition and must be dissolved.

Britain's problem is to find an acceptable course reconciling the two conflicting demands.

The Patriotic Front and its supporters among neighboring African states have no illusions that Britain may take a position favoring the guerrillas in any way.

George Silundika, Nkomo's foreign policy expert, said the British are likely to tilt toward Muzorewa and seek simply to expand his government to handle the transition. Such a solution, he said, would be unacceptable to the Front.

"I don't see my way through this conference. I would like to be convinced that the British government will play ball with us, but I'm not," he said.

Zimbabwe-Rhodesian officials are hoping that fear of backlash at the Conservative Party conference starting Oct. 9 will force Prime Minister Margaret Thatcher's government to support Muzorewa on transitional arrangements.

The officials ruefully admit, however, that hopes for Western support in the past invariably have been dashed. The latest letdown for Muzorewa came yesterday when the U.S. Senate backed off from an effort to force President Carter to lift economic sanctions against Zimbabwe-Rhodesia immediately.

The next big target of hope for Salisbury, is Nov. 15., the deadline for the British Parliament to renew sanctions. By then, however, the conference is expected to have either settled the transitional matters or be headed for failure.

Britain's strong card is that neither side can afford to be seen as the wrecker of the conference. The alternative to reaching a peaceful settlement in London is an escalation of the war, which has already killed more than 20,000 people, and the potential for East-West involvement.

African states supporting the guerrillas and South Africa -- which bankrolls much of the Zimbabwe-Rhodesia's daily war effort costing $1.6 million -- are pressing their clients to reach agreement. Both supporting sides, however, only want agreement if it favors the horse they are backing.

Perhaps the best chance of reaching a settlement in which Britain would supervise new elections lies in the confidence of both the Muzorewa and Mugabe-Nkomo camps that they would win the vote.

Somebody has to be wrong. The next few weeks could well determine whether the voters ever get a chance to say who is right.